introduce global greenpool

This change add a global greenpool which is used to manage
the greenthreads created via nova.utils.spawn(_n).

A test fixture is also added to use an isolated greenpool
which will raise an exception if a greenthread is leaked.
the fixture will optionally raise if greenlets are leaked.
This is enabled by unit test by default and is configurable
for functional tests.

This change removes all greenthread leaks from the unit
and functional tests that were detected. 7 functional
tests still leak greenlets but they have no obvious
cause. as such greenlet leaks are not treated as errors
for funtional tests by default. Greenthread leaks
are always treated as errors.
Set NOVA_RAISE_ON_GREENLET_LEAK=1|true|yes when invoking
tox to make greenlet leaks an error for functional tests.

Change-Id: I73b4684744b340bfb80da08537a745167ddea106

tox.ini

@@ -30,6 +30,16 @@ extras =
 passenv =
   OS_DEBUG
   GENERATE_HASHES
+# Note(sean-k-mooney):
+# leaking greanthreads between tests is a bug so we should fail if it happens
+# however unlike greenthreads, greenlets cannot be killed via a test fixture.
+# greenlet leaks will be annotated in the test details but will not cause a
+# failure. if you want to make them raise set
+# NOVA_RAISE_ON_GREENLET_LEAK=1|true|yes
+# All simiple leaks of green threads have been resolved the remaining 7
+# functional test failures where greenlet leaks happen are non trivial
+# to address as there is no obvious cause so they are ignored for now.
+  NOVA_RAISE_ON_GREENLET_LEAK
 # NOTE(sean-k-mooney) optimization is enabled by default and when enabled
 # asserts are complied out. Disable optimization to allow asserts in
 # nova to fire in unit and functional tests. This can be useful for
@@ -42,6 +52,13 @@ commands =
   env TEST_OSPROFILER=1 stestr run --combine --no-discover 'nova.tests.unit.test_profiler'
   stestr slowest
 
+[testenv:{unit,py3,py38,py39,py310,py311}]
+setenv =
+  {[testenv]setenv}
+# we do not have any greenlet leaks in unit tests so enforce that
+# by making greenlet leaks a failure.
+  NOVA_RAISE_ON_GREENLET_LEAK=True
+
 [testenv:functional{,-py38,-py39,-py310,-py311}]
 description =
   Run functional tests.