This patch switches the default concurrency mode to native threading
for the services that gained native threading support in Flamingo:
nova-scheduler, nova-api, and nova-metadata.
The OS_NOVA_DISABLE_EVENTLET_PATCHING env variable still can be used to
explicitly switch the concurrency mode to eventlet by
OS_NOVA_DISABLE_EVENTLET_PATCHING=false
We also ensure that the cover, docs, py3xx and functional tox targets
are still running with eventlet while py312-threading kept running
with native threading.
Change-Id: I86c7f31f19ca3345218171f0abfa8ddd4f8fc7ea
Signed-off-by: Balazs Gibizer <gibi@redhat.com>
The end goals is to be able to run at least some of the unit tests
without eventlet. But there are things preventing that for now.
We need to make sure that the oslo.sevice backed is not initialized to
eventlet by any early import code before our monkey_patch module can do
the selective backed selection based on the env variable.
The nova.tests.unit module had some import time code execution that is
forcing imports that initialize the oslo.service backend too early,
way before nova would do it in normal execution. We could remove
objects.register_all() from nova/tests/unit/__init__.py as it seems
tests are passing without it. Still that would not be enough so I
eventually decide to keep it.
The other issue is that the unit test discovery imports all modules
under nova.tests.unit and that eventually imports oslo.messaging and
that also forces oslo.service backend selection.
So we injected an early call to our smart monkey_patch module to preempt
that. This does not change the imported modules as monkey_patch module
imported anyhow via nova.test module. Just changed the order to allow
oslo.service backend selection explicitly.
After this patch the unit test can be run via
OS_NOVA_DISABLE_EVENTLET_PATCHING=true tox -e py312
Most of the test will pass but there are a bunch of test timing out or
hanging.
Change-Id: I210cb6a30deaee779d55f88f0f57584c65b0dc05
Signed-off-by: Balazs Gibizer <gibi@redhat.com>
This patch refines our logging, doc, and release notes about the native
threading mode of scheduler, api, and metadata services to ask for
pre-prod testing before enabled in production.
Change-Id: I04bbb3d7e4664a0cab8b30f4c34ee71774536353
Signed-off-by: Balazs Gibizer <gibi@redhat.com>
At the service startup nova need to initialize either the eventlet or
the threading backend of oslo.service. So this patch reuses the existing
logic behind OS_NOVA_DISABLE_EVENTLET_PATCHING.
When OS_NOVA_DISABLE_EVENTLET_PATCHING env variable is set to true the
service will select the threading backend otherwise the eventlet
backend.
Also to avoid later monkey patch calls to invalidated the selection if
the threading backend is selected then the monkey_patch code is
poisoned.
This patch also makes sure that oslo.messaging also initialized with the
matching executor backend.
As this is the last step to make nova-scheduler run in threading mode
this patch adds a release notes as well.
Change-Id: I6e2e6a43df78d23580b5e7402352a5036100ab36
Signed-off-by: Balazs Gibizer <gibi@redhat.com>
This change separates the evetlet monkey patching
from importing the module and add a module level
constant to track if we have already monkey patched.
Change-Id: Ic4ab0ba7a8320a008d6e246641446446dcc9ccc0
The nova debuger functionality was intended
to help debugging running process however it has
never been reliable due to our use of eventlet and is generally
not required when not using eventlet. I.e. you can just
run the nova console-scripts form a debugger or add pdb
statements as required.
As part of the eventlet removal the debugger functionality is
removed given its untested and undocumented.
Change-Id: I7bf88f06f3d1dbd2c7e342b27a21440a123c631d
Since Hyper-V driver was removed, we don't expect any use case where
nova should be deployed in Windows OS. This scenario is no longer
tested since WinStackers project has been retired.
Drop the remaining code for Windows OS support to reduce unused code
and focus purely on linux systems.
Change-Id: I69b049d13556efe87f4be32f5e6e7f88e42f3b2a
This code worked around a bug in eventlet[1] that has been fixed in
6533958f5fcec8b1577f56834ea2dc75a4f211e3. The fix has been available in
every eventlet release since v0.27.0.
[1] https://github.com/eventlet/eventlet/issues/592
Co-Authored-By: Cyril Roelandt <cyril@redhat.com>
Change-Id: I82e8a27fd82906cf068f5107c898ad660c826d7d
Back in the days of centos 6 and python 2.6 eventlet
greendns monkeypatching broke ipv6. As a result nova
has run without greendns monkey patching ever since.
This removes that old workaround allowing modern
eventlet to use greendns for non blocking dns lookups.
Closes-Bug: #1964149
Change-Id: Ia511879d2f5f50a3f63d180258abccf046a7264e
Previously, we were setting the environment variable to disable
greendns in eventlet *after* import eventlet. This has no effect, as
eventlet processes environment variables at import time. This patch
moves the setting of EVENTLET_NO_GREENDNS before importing eventlet in
order to correctly disable greendns.
Closes-bug: 1895322
Change-Id: I4deed815c8984df095019a7f61d089f233f1fc66
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: I4872169413f27aeaff8d8fdfa5cdaf6ee32f4680
Closes-Bug: #1863021
This brings in a couple of new checks which must be addressed, many of
which involve a rather large amount of changes, so these are ignored for
now. A series of follow-up changes will resolved these.
'pycodestyle' is added as a dependency rather than it being pulled in
transitively. This is necessary since we're using it in tests.
Change-Id: I35c654bd39f343417e0a1124263ff31dcd0b05c9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We were seeing infinite recursion opening an ssl socket when running
various combinations of python3, eventlet, and urllib3. It is not
clear exactly what combination of versions are affected, but for
background there is an example of this issue documented here:
https://github.com/eventlet/eventlet/issues/371
The immediate cause in nova's case was that we were calling
eventlet.monkey_patch() after importing urllib3. Specifically, change
Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
nova.utils.monkey_patch() method to make monkey patching common
between WSGI and non-WSGI services. Unfortunately, before executing
this method you must first import nova.utils, which imports a large
number of modules itself. Anything imported (transitively) by
nova.utils would therefore be imported before monkey patching, which
included urllib3. This triggers the infinite recursion problem
described above if you have an affected combination of library
versions.
While this specific issue may eventually be worked around or fixed in
eventlet or urllib3, it remains true that eventlet best practises are
to monkey patch as early as possible, which we were not doing. To
avoid this and hopefully future similar issues, this change ensures
that monkey patching happens as early as possible, and only a minimum
number of modules are imported first.
This change fixes monkey patching for both non-wsgi and wsgi callers:
* Non-WSGI services (nova/cmd)
This is fixed by using the new monkey_patch module, which has minimal
dependencies.
* WSGI services (nova/api/openstack)
This is fixed both by using the new monkey_patch module, and by moving
the patching point up one level so that it is done before importing
anything in nova/api/openstack/__init__.py.
This move causes issues for some external tools which load this path
from nova and now monkey patch where they previously did not. However,
it is unfortunately unavoidable to enable monkey patching for the wsgi
entry point without major restructuring. This change includes a
workaround for sphinx to avoid this issue.
This change has been through several iterations. I started with what
seemed like the simplest and most obvious change, and moved on as I
discovered more interactions which broke. It is clear that eventlet
monkey patching is extremely fragile, especially when done implicitly at
module load time as we do. I would advocate a code restructure to
improve this situation, but I think the time would be better spent
removing the eventlet dependency entirely.
Co-authored-by: Lee Yarwood <lyarwood@redhat.com>
Closes-Bug: #1808975
Closes-Bug: #1808951
Change-Id: Id46e76666b553a10ec4654d4418a9884975b5b95
Balazs Gibizer