The commit 887c445a7a made the nova.utils
module dependent on the fixtures library but the change missed updating
requirements and the fixtures library is not installed automatically.
This change migrates the fixtures library from test-requirements.txt to
requirements.txt so that the library is installed without test codes.
Closes-Bug: #1958075
Change-Id: I712f88fc1b6053fe6d1f13e708f3bd8874452a8f
Some things that were missed in previous patches and are thrown together
here:
- Add alembic as an explicit dependency (we were getting it transitively
from oslo.db). We also bump the sqlalchemy dependency to a 1.4.x
release, which is the minimum supported by our chosen version of
alembic (more on this below)
- Remove tooling related to the old migrations
- Fix the tox whitelisting of the flaky MySQL tests
On the SQLAlchemy front, we opt for 1.4.13. Technically alembic should
support anything from 1.4.0, however, with SQLAlchemy >= 1.4.0, < 1.4.13
we see errors like the following in some tests:
sqlalchemy.exc.InvalidRequestError: Entity namespace for
"count(instance_mappings.id)" has no property "queued_for_delete"
There's nothing specific about this in the release notes for 1.4.13 [1]
but it definitely fixes things.
[1] https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.13
Change-Id: I4c8eb13f11aa7471c26a5ba326319aef245c9836
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This makes it possible to partially revert
If48dd949ec4d69a09c87178f16d56a2517e21fd8 as one of the warning silenced
there now fixed in oslo.db.
Change-Id: I0051c0f8694699865bc16636953908d60ba456d2
Take advantage of the neutronclient bindings for the port binding APIs
added in neutronclient 7.1.0 to avoid having to vendor this stuff
ourselves.
Change-Id: Icc284203fb53658abe304f24a62705217f90b22b
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Since 3.7.0, oslo policy started the DeprecationWarning[1] if
deprecated_reason and deprecated_since param are not passed
in DeprecatedRule or they are passed in RuleDefault object.
[1] https://github.com/openstack/oslo.policy/blob/3.7.0/oslo_policy/policy.py#L1538
Change-Id: Idbbc203c6ae65aee29f9463a4911bae2bb541f41
Use the new oslo.confg type HostDomainOpt to support underscore in the
name. You can see the bugzilla[1] to have more information.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1868940
Related-Bug: #1892044
Change-Id: Ib6c8fc1e3d90b79b10066c429670fcb957bddd23
os-vif 1.15.0 added the ability to create an OVS port during plugging
by specifying the 'create_port' attribute in the 'port_profile' field.
By delegating port creation to os-vif, we can rely on it's 'isolate_vif'
config option [1] that will temporarily configure the VLAN to 4095
(0xfff), which is reserved for implementation use [2] and is used by
neutron to as a dead VLAN [3]. By doing this, we ensure VIFs are plugged
securely, preventing guests from accessing other tenants' networks
before the neutron OVS agent can wire up the port.
This change requires a little dance as part of the live migration flow.
Since we can't be certain the destination host has a version of os-vif
that supports this feature, we need to use a sentinel to indicate when
it does. Typically we would do so with a field in
'LibvirtLiveMigrateData', such as the 'src_supports_numa_live_migration'
and 'dst_supports_numa_live_migration' fields used to indicate support
for NUMA-aware live migration. However, doing this prevents us
backporting this important fix since o.vo changes are not backportable.
Instead, we (somewhat evilly) rely on the free-form nature of the
'VIFMigrateData.profile_json' string field, which stores JSON blobs and
is included in 'LibvirtLiveMigrateData' via the 'vifs' attribute, to
transport this sentinel. This is a hack but is necessary to work around
the lack of a free-form "capabilities" style dict that would allow us do
backportable fixes to live migration features.
Note that this change has the knock on effect of modifying the XML
generated for OVS ports: when hybrid plug is false will now be of type
'ethernet' rather than 'bridge' as before. This explains the larger than
expected test damage but should not affect users.
[1] https://opendev.org/openstack/os-vif/src/tag/2.4.0/vif_plug_ovs/ovs.py#L90-L93
[2] https://en.wikipedia.org/wiki/IEEE_802.1Q#Frame_format
[3] https://answers.launchpad.net/neutron/+question/231806
Change-Id: I11fb5d3ada7f27b39c183157ea73c8b72b4e672e
Depends-On: Id12486b3127ab4ac8ad9ef2b3641da1b79a25a50
Closes-Bug: #1734320
Closes-Bug: #1815989
The os-brick 4.3.1 release fixes a critical data loss bug that can be
triggered by reconfiguring [libvirt]/volume_use_multipath while volumes
are attached.
The following requirements and lower-requirements are also bumped as
introduced into os-brick by Ie7dcc859e1291bd1d50b1ae516be38d8732de836:
os-brick 4.3.1 depends on eventlet>=0.30.1
os-brick 4.3.1 depends on oslo.concurrency>=4.4.0
os-brick 4.3.1 depends on oslo.service>=2.5.0
os-brick 4.3.1 depends on oslo.serialization>=4.1.0
os-brick 4.3.1 depends on oslo.utils>=4.8.0
os-brick 4.3.1 depends on pbr>=5.5.1
os-brick 4.3.1 depends on requests>=2.25.1
os-brick 4.3.1 depends on tenacity>=6.3.1
Co-Authored-By: Lee Yarwood <lyarwood@redhat.com>
Related-Bug: #1921381
Change-Id: I0d5df271dec1c6035229b02ec45e03abad6f27e7
Ceph 16 (Pacific) will support attaching RBD images to Windows
hosts as well as Hyper-V VMs [1].
This patch updates the Hyper-V driver so that it may be able to
consume RBD volumes.
We're also updating the os-win requirement to 5.4.0. It should've
been bumped by [2], vms won't spawn without it. Note this introduces no
new or unmet direct or in-direct version dependencies and so is included
as part of this change.
[1] https://github.com/ceph/ceph/pull/33750
[2] Ibe6aff4edeb32208bc9865e9216a7432caddab2b
Implements: blueprint hyperv-rbd
Change-Id: I9ad90817648ca12f80a6b53f6ba728df15cbafab
This change simply bumps the version of os-brick required by Nova to
version 4.2.0. This is itself required by
I9ad90817648ca12f80a6b53f6ba728df15cbafab that introduces support for
rbd volumes within the HyperV driver.
After much back and fourth it was decided to bump the required version
of os-brick ahead of that change to allow the following requirements.txt
and lower-constraints.txt changes caused by direct and in-direct
dependency changes to be documented clearly:
os-brick 4.2.0 depends on oslo.log>=4.4.0
\_ oslo-log 4.4.0 depends on python-dateutil>=2.7.0
os-brick 4.2.0 depends on oslo.serialization>=4.0.1
os-brick 4.2.0 depends on pbr>=5.5.0
os-brick 4.2.0 depends on oslo.privsep>=2.4.0
\_ oslo.privsep 2.4.0 depends on msgpack>=0.6.0
os-brick 4.2.0 depends on oslo.service>=2.4.0
os-brick 4.2.0 depends on tenacity>=6.2.0
os-brick 4.2.0 depends on oslo.context>=3.1.1
os-brick 4.2.0 depends on oslo.concurrency>=4.3.0
os-brick 4.2.0 depends on oslo.i18n>=5.0.1
os-brick 4.2.0 depends on six>=1.15.0
os-brick 4.2.0 depends on os-win>=5.1.0
The above changes have been tested with pip 21.0.1 to ensure the new
resolver is happy and that nothing has been missed.
Change-Id: Ic83f3c7c955d0df89d75f700ee4fe2bd7f715794
Implements: blueprint hyperv-rbd
Since only Wallaby compute nodes will support the 'socket' PCI NUMA
affinity policy, this patch adds a ResourceRequest translator that adds
a required trait if the value of '(hw_|hw:)pci_numa_affinity_policy' is
'socket'.
The actual trait reporting by the libvirt driver will be added in a
future patch. Until then the 'socket' value remains a hidden no-op.
Implements: blueprint pci-socket-affinity
Change-Id: I908ff07e1107304ca5926cc04d2fdc8fe0da5ed9
Replace six.text_type with str.
This patch completes six removal.
Change-Id: I779bd1446dc1f070fa5100ccccda7881fa508d79
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
The dnspython 2.0.0 incompatibility[1] is worked around in eventlet 0.26.1
by restricting the dnspython version to < 2.0.0. So the issue affects
nova-novncproxy as well we need to bump our lower constraint as well.
[1] https://github.com/eventlet/eventlet/issues/619
Related-Bug: #1888237
Change-Id: Ie9cc718e654cb37a4f31ccb4b052b23878d476a2
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() has been added to
oslo_utils. See https://review.opendev.org/#/c/750031/
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
The instances being replaced here appear to be used to provide
representations for paths. There is in fact already a sha256 version
of get_hash_str that is supposed to be used in security sensitive
usages.
With this change (and the related dependent changes), the unit and
functional tests pass when run on a FIPS enabled system.
Change-Id: If0ec11e7b7fcde4dacc57265c4dd77b0f536bfab
Depends-On: https://review.opendev.org/#/c/756432
Depends-On: https://review.opendev.org/#/c/756153
Depends-On: https://review.opendev.org/#/c/760160
Default value of 'CONF.oslo_policy.policy_file' config option
has been changed from 'policy.json' to 'policy.yaml'. If new default
file 'policy.yaml' does not exist but old default 'policy.json' exist
then fallback to use old default file.
An upgrade checks is added to check the policy_file format and
fail upgrade checks if it is JSON formatted.
Added a warning in policy doc about JSON formatted file is deprecated,
also removed all the reference to policy.json file in doc as well as
in tests.
Related Blueprint: policy-json-to-yaml
Closes-Bug: #1875418
Change-Id: Ic4d3b998bb9701cb1e3ef12d9bb6f4d91cc19c18
Several version specified in our lower-constraints files had conflicting
dependencies. This updates a few packages to avoid those conflicts. It
also removes the linters that are tracked in the global requirements
blocklist since we do not need to enforce lower-constraints for linters.
Change-Id: Iae93f0631d77794f093fd330e849d2fd805b238f
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This series implements the referenced blueprint to allow for specifying
custom resource provider traits and inventories via yaml config files.
This first commit includes schema version 1.0 and functions to load and
validate against that schema.
Since this patch, package ddt>=1.2.1 is required to run tests.
Co-Author: Tony Su <tao.su@intel.com>
Author: Dustin Cowles <dustin.cowles@intel.com>
Blueprint: provider-config-file
Change-Id: I58099726f799c427e9174a0dcce9889344d51e7c
To support new AVX512 cpu feature, this patch adds the mapping
between cpu features reported by libvirt [1] and corresponding
os-traits [2]. The traits include 'AVX512VBMI', 'AVX512IFMA',
'AVX512VBMI2', 'AVX512BITALG', 'AVX512VAES', 'AVX512GFNI',
'AVX512VPCLMULQDQ' and 'AVX512VPOPCNTDQ'.
[1] https://github.com/libvirt/libvirt/blob/master/src/cpu_map/x86_features.xml
[2] https://review.opendev.org/#/c/728576/
Depends-On: I58cba02f204d2e19a1a838852c628694edd47d73
Change-Id: I212a84e5cae15bc133aafd538d55f85149ab360f
oslo has new flag to disable the warning for default check_str
change and keep logging warning if policy name is changed or it is
marked for removal.
Policy defaults refresh BP work changed the defaults for all the
policy so logging their warning is dangerousa and fill up the log file.
Once we switch to new defaults then we can enable them if needed.
Logging warning for name change and marked for removal make sense
when old rules are override by operator so the new flag does not control
those. For policy defaults refresh BP work, we do not have many policies
in this category so logging their warning is ok.
This commits reset the testing flag 'suppress_deprecation_warnings'
default value back to False. Test policy fixture sets that to True to
avoid any warnig during tests run.
Bumping the oslo policy version where this new flag is present
- https://review.opendev.org/#/c/719547/
Partial implement blueprint policy-defaults-refresh
Change-Id: Iaa6baf6877890babb6205bdb3ee2e1a2b28ebd9c
Add the validation framework necessary to verify extra specs along with
the definitions for every extra spec we currently recognize in-tree.
None of this is currently used since we don't have the API microversions
wired up, but that will come in a future patch.
Note that we must add the H238 hacking check to the ignore list here,
since this includes our first use of Python 3-type classes without the
explicit 'object' subclass. This can be removed when that check is
removed from hacking.
Part of blueprint flavor-extra-spec-validators
Change-Id: Ib64a1348cce1dca995746214616c4f33d9d664bd
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Building on the ``[workarounds]/disable_native_luksv1``
configurable introduced in Ia500eb614cf575ab846f64f4b69c9068274c8c1f
this change introduces another workaround configurable that when enabled
will connect RBD volumes to the compute host as block devices using
os-brick.
When used togther both options allow operators to workaround recently
discovered performance issues in the libgcrypt library used by QEMU when
natively decrypting LUKSv1 encrypted disks.
For now the extend_volume method raises a NotImplemented error in-line
with the underlying method in os-brick. Future work will be required to
both support this in os-brick and wire up the required calls in the
volume driver.
This workaround is temporary and will be removed during the W release
once all impacted distributions have been able to update their versions
of the libgcrypt library.
Finally os-brick 3.0.1 is now required as it provides the
Id507109df80391699074773f4787f74507c4b882 fix when attempting to
diconnect locally attached RBD volumes.
Closes-Bug: #1869184
Change-Id: Ied3732042738a6194b635c55e0304d71a6fb66e3
This change is a follow up to I0c3f14100a18107f7e416293f3d4fcc641ce5e55
and removes the direct call to nova.privsep.qemu with one to the images
API that now returns an oslo_utils.imageutils.QemuImgInfo object.
Version 4.1.0 of oslo.utils introducing support for the format-specific
data returned by qemu-img info for LUKSv1 based images.
Change-Id: I573396116e10cf87f80f1ded55f2cd8f498859e4
When the resource tracker has to lock a compute host for updates or
inspection, it uses a single semaphore. In most cases, this is fine, as
a compute process only is tracking one hypervisor. However, in Ironic, it's
possible for one compute process to track many hypervisors. In this
case, wait queues for instance claims can get "stuck" briefly behind
longer processing loops such as the update_resources periodic job. The
reason this is possible is because the oslo.lockutils synchronized
library does not use fair locks by default. When a lock is released, one
of the threads waiting for the lock is randomly allowed to take the lock
next. A fair lock ensures that the thread that next requested the lock
will be allowed to take it.
This should ensure that instance claim requests do not have a chance of
losing the lock contest, which should ensure that instance build
requests do not queue unnecessarily behind long-running tasks.
This includes bumping the oslo.concurrency dependency; fair locks were
added in 3.29.0 (I37577becff4978bf643c65fa9bc2d78d342ea35a).
Change-Id: Ia5e521e0f0c7a78b5ace5de9f343e84d872553f9
Related-Bug: #1864122
This includes a couple of fixes and removes the need for custom
workarounds, like the removal of the 'address_string' helper at [1].
[1] https://github.com/novnc/websockify/commit/be9823bf
Change-Id: I9b1d57fd513386e552afbbc8a59cf3a4c54b6908
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This uses the COMPUTE_SAME_HOST_COLD_MIGRATE trait in the API during a
cold migration to filter out hosts that cannot support same-host cold
migration, which is all of them except for the hosts using the vCenter
driver.
For any nodes that do not report the trait, we won't know if they don't
because they don't support it or if they are not new enough to report
it, so the API has a service version check and will fallback to old
behavior using the config if the node is old. That compat code can be
removed in the next release.
As a result of this the FakeDriver capabilities are updated so the
FakeDriver no longer supports same-host cold migration and a new fake
driver is added to support that scenario for any tests that need it.
Change-Id: I7a4b951f3ab324c666ab924e6003d24cc8e539f5
Closes-Bug: #1748697
Related-Bug: #1811235
This reverts commit 33a2a5f2e4. We're now
at M1 and the DevStack default has been changed. We no longer need to
pretend to support Python 2.7. This is slightly modified to drop an
additional Python 2.7-only dependency from lower-constraints.txt.
[1] I52b03caee0ba700da3a15035201ea6cd91baa06b
Change-Id: I1068850072e639ee268d68ca82249f569a40310e
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Revert part of the changes from
Ie1a0cbd82a617dbcc15729647218ac3e9cd0e5a9
This way it is possible again for nova to be installed
using python 2, thus allowing other devstack-based
jobs to work until everything switches to python 3
according the plans.
While py2 support may break anytime, its days are limited anyway.
Change-Id: I99ff84e6c1c033c28d499b0b8ac17ac5b0565f61
It's Ussuri. We can *finally* stop testing Python 2 [1]. Time to party.
We don't attempt any cleanup but simply stop testing with Python 2,
indicate that we only support Python 3 via 'setup.cfg' and remove any
Python 2 only dependencies. Our 'tox.ini' is modified such that
'functional' now runs with 'python3', whatever that may point to, though
the gate will only use a versioned variant (currently
'functional-py36').
This should free up a significant amount of resources from the gate and
let us start using Python 3 idioms in our code. Win-win.
[1] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html#python2-deprecation-timeline
Change-Id: Ie1a0cbd82a617dbcc15729647218ac3e9cd0e5a9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Takashi Kajinami