Lee Yarwood
dd1e6d4b0c
Bug #1894804 outlines how DEVICE_DELETED events were often missing from QEMU on Focal based OpenStack CI hosts as originally seen in bug #1882521. This has eventually been tracked down to some undefined QEMU behaviour when a new device_del QMP command is received while another is still being processed, causing the original attempt to be aborted. We hit this race in slower OpenStack CI envs as n-cpu rather crudely retries attempts to detach devices using the RetryDecorator from oslo.service. The default incremental sleep time currently being tight enough to ensure QEMU is still processing the first device_del request on these slower CI hosts when n-cpu asks libvirt to retry the detach, sending another device_del to QEMU hitting the above behaviour. Additionally we have also seen the following check being hit when testing with QEMU >= v5.0.0. This check now rejects overlapping device_del requests in QEMU rather than aborting the original: https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b This change aims to avoid this situation entirely by raising the default incremental sleep time between detach requests from 2 seconds to 10, leaving enough time for the first attempt to complete. The overall maximum sleep time is also increased from 30 to 60 seconds. Future work will aim to entirely remove this retry logic with a libvirt event driven approach, polling for the the VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED and VIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED events before retrying. Finally, the cleanup of unused arguments in detach_device_with_retry is left for a follow up change in order to keep this initial change small enough to quickly backport. Closes-Bug: #1882521 Related-Bug: #1894804 Change-Id: Ib9ed7069cef5b73033351f7a78a3fb566753970d
==============
OpenStack Nova
==============
.. image:: https://governance.openstack.org/tc/badges/nova.svg
:target: https://governance.openstack.org/tc/reference/tags/index.html
.. Change things from this point on
OpenStack Nova provides a cloud computing fabric controller, supporting a wide
variety of compute technologies, including: libvirt (KVM, Xen, LXC and more),
Hyper-V, VMware, OpenStack Ironic and PowerVM.
Use the following resources to learn more.
API
---
To learn how to use Nova's API, consult the documentation available online at:
- `Compute API Guide <https://docs.openstack.org/api-guide/compute/>`__
- `Compute API Reference <https://docs.openstack.org/api-ref/compute/>`__
For more information on OpenStack APIs, SDKs and CLIs in general, refer to:
- `OpenStack for App Developers <https://www.openstack.org/appdev/>`__
- `Development resources for OpenStack clouds
<https://developer.openstack.org/>`__
Operators
---------
To learn how to deploy and configure OpenStack Nova, consult the documentation
available online at:
- `OpenStack Nova <https://docs.openstack.org/nova/>`__
In the unfortunate event that bugs are discovered, they should be reported to
the appropriate bug tracker. If you obtained the software from a 3rd party
operating system vendor, it is often wise to use their own bug tracker for
reporting problems. In all other cases use the master OpenStack bug tracker,
available at:
- `Bug Tracker <https://bugs.launchpad.net/nova>`__
Developers
----------
For information on how to contribute to Nova, please see the contents of the
CONTRIBUTING.rst.
Any new code must follow the development guidelines detailed in the HACKING.rst
file, and pass all unit tests.
Further developer focused documentation is available at:
- `Official Nova Documentation <https://docs.openstack.org/nova/>`__
- `Official Client Documentation
<https://docs.openstack.org/python-novaclient/>`__
Other Information
-----------------
During each `Summit`_ and `Project Team Gathering`_, we agree on what the whole
community wants to focus on for the upcoming release. The plans for nova can
be found at:
- `Nova Specs <http://specs.openstack.org/openstack/nova-specs/>`__
.. _Summit: https://www.openstack.org/summit/
.. _Project Team Gathering: https://www.openstack.org/ptg/