This change simply bumps the version of os-brick required by Nova to
version 4.2.0. This is itself required by
I9ad90817648ca12f80a6b53f6ba728df15cbafab that introduces support for
rbd volumes within the HyperV driver.
After much back and fourth it was decided to bump the required version
of os-brick ahead of that change to allow the following requirements.txt
and lower-constraints.txt changes caused by direct and in-direct
dependency changes to be documented clearly:
os-brick 4.2.0 depends on oslo.log>=4.4.0
\_ oslo-log 4.4.0 depends on python-dateutil>=2.7.0
os-brick 4.2.0 depends on oslo.serialization>=4.0.1
os-brick 4.2.0 depends on pbr>=5.5.0
os-brick 4.2.0 depends on oslo.privsep>=2.4.0
\_ oslo.privsep 2.4.0 depends on msgpack>=0.6.0
os-brick 4.2.0 depends on oslo.service>=2.4.0
os-brick 4.2.0 depends on tenacity>=6.2.0
os-brick 4.2.0 depends on oslo.context>=3.1.1
os-brick 4.2.0 depends on oslo.concurrency>=4.3.0
os-brick 4.2.0 depends on oslo.i18n>=5.0.1
os-brick 4.2.0 depends on six>=1.15.0
os-brick 4.2.0 depends on os-win>=5.1.0
The above changes have been tested with pip 21.0.1 to ensure the new
resolver is happy and that nothing has been missed.
Change-Id: Ic83f3c7c955d0df89d75f700ee4fe2bd7f715794
Implements: blueprint hyperv-rbd
Since only Wallaby compute nodes will support the 'socket' PCI NUMA
affinity policy, this patch adds a ResourceRequest translator that adds
a required trait if the value of '(hw_|hw:)pci_numa_affinity_policy' is
'socket'.
The actual trait reporting by the libvirt driver will be added in a
future patch. Until then the 'socket' value remains a hidden no-op.
Implements: blueprint pci-socket-affinity
Change-Id: I908ff07e1107304ca5926cc04d2fdc8fe0da5ed9
Replace six.text_type with str.
This patch completes six removal.
Change-Id: I779bd1446dc1f070fa5100ccccda7881fa508d79
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
The dnspython 2.0.0 incompatibility[1] is worked around in eventlet 0.26.1
by restricting the dnspython version to < 2.0.0. So the issue affects
nova-novncproxy as well we need to bump our lower constraint as well.
[1] https://github.com/eventlet/eventlet/issues/619
Related-Bug: #1888237
Change-Id: Ie9cc718e654cb37a4f31ccb4b052b23878d476a2
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() has been added to
oslo_utils. See https://review.opendev.org/#/c/750031/
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
The instances being replaced here appear to be used to provide
representations for paths. There is in fact already a sha256 version
of get_hash_str that is supposed to be used in security sensitive
usages.
With this change (and the related dependent changes), the unit and
functional tests pass when run on a FIPS enabled system.
Change-Id: If0ec11e7b7fcde4dacc57265c4dd77b0f536bfab
Depends-On: https://review.opendev.org/#/c/756432
Depends-On: https://review.opendev.org/#/c/756153
Depends-On: https://review.opendev.org/#/c/760160
Default value of 'CONF.oslo_policy.policy_file' config option
has been changed from 'policy.json' to 'policy.yaml'. If new default
file 'policy.yaml' does not exist but old default 'policy.json' exist
then fallback to use old default file.
An upgrade checks is added to check the policy_file format and
fail upgrade checks if it is JSON formatted.
Added a warning in policy doc about JSON formatted file is deprecated,
also removed all the reference to policy.json file in doc as well as
in tests.
Related Blueprint: policy-json-to-yaml
Closes-Bug: #1875418
Change-Id: Ic4d3b998bb9701cb1e3ef12d9bb6f4d91cc19c18
Several version specified in our lower-constraints files had conflicting
dependencies. This updates a few packages to avoid those conflicts. It
also removes the linters that are tracked in the global requirements
blocklist since we do not need to enforce lower-constraints for linters.
Change-Id: Iae93f0631d77794f093fd330e849d2fd805b238f
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This series implements the referenced blueprint to allow for specifying
custom resource provider traits and inventories via yaml config files.
This first commit includes schema version 1.0 and functions to load and
validate against that schema.
Since this patch, package ddt>=1.2.1 is required to run tests.
Co-Author: Tony Su <tao.su@intel.com>
Author: Dustin Cowles <dustin.cowles@intel.com>
Blueprint: provider-config-file
Change-Id: I58099726f799c427e9174a0dcce9889344d51e7c
To support new AVX512 cpu feature, this patch adds the mapping
between cpu features reported by libvirt [1] and corresponding
os-traits [2]. The traits include 'AVX512VBMI', 'AVX512IFMA',
'AVX512VBMI2', 'AVX512BITALG', 'AVX512VAES', 'AVX512GFNI',
'AVX512VPCLMULQDQ' and 'AVX512VPOPCNTDQ'.
[1] https://github.com/libvirt/libvirt/blob/master/src/cpu_map/x86_features.xml
[2] https://review.opendev.org/#/c/728576/
Depends-On: I58cba02f204d2e19a1a838852c628694edd47d73
Change-Id: I212a84e5cae15bc133aafd538d55f85149ab360f
oslo has new flag to disable the warning for default check_str
change and keep logging warning if policy name is changed or it is
marked for removal.
Policy defaults refresh BP work changed the defaults for all the
policy so logging their warning is dangerousa and fill up the log file.
Once we switch to new defaults then we can enable them if needed.
Logging warning for name change and marked for removal make sense
when old rules are override by operator so the new flag does not control
those. For policy defaults refresh BP work, we do not have many policies
in this category so logging their warning is ok.
This commits reset the testing flag 'suppress_deprecation_warnings'
default value back to False. Test policy fixture sets that to True to
avoid any warnig during tests run.
Bumping the oslo policy version where this new flag is present
- https://review.opendev.org/#/c/719547/
Partial implement blueprint policy-defaults-refresh
Change-Id: Iaa6baf6877890babb6205bdb3ee2e1a2b28ebd9c
Add the validation framework necessary to verify extra specs along with
the definitions for every extra spec we currently recognize in-tree.
None of this is currently used since we don't have the API microversions
wired up, but that will come in a future patch.
Note that we must add the H238 hacking check to the ignore list here,
since this includes our first use of Python 3-type classes without the
explicit 'object' subclass. This can be removed when that check is
removed from hacking.
Part of blueprint flavor-extra-spec-validators
Change-Id: Ib64a1348cce1dca995746214616c4f33d9d664bd
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Building on the ``[workarounds]/disable_native_luksv1``
configurable introduced in Ia500eb614cf575ab846f64f4b69c9068274c8c1f
this change introduces another workaround configurable that when enabled
will connect RBD volumes to the compute host as block devices using
os-brick.
When used togther both options allow operators to workaround recently
discovered performance issues in the libgcrypt library used by QEMU when
natively decrypting LUKSv1 encrypted disks.
For now the extend_volume method raises a NotImplemented error in-line
with the underlying method in os-brick. Future work will be required to
both support this in os-brick and wire up the required calls in the
volume driver.
This workaround is temporary and will be removed during the W release
once all impacted distributions have been able to update their versions
of the libgcrypt library.
Finally os-brick 3.0.1 is now required as it provides the
Id507109df80391699074773f4787f74507c4b882 fix when attempting to
diconnect locally attached RBD volumes.
Closes-Bug: #1869184
Change-Id: Ied3732042738a6194b635c55e0304d71a6fb66e3
This change is a follow up to I0c3f14100a18107f7e416293f3d4fcc641ce5e55
and removes the direct call to nova.privsep.qemu with one to the images
API that now returns an oslo_utils.imageutils.QemuImgInfo object.
Version 4.1.0 of oslo.utils introducing support for the format-specific
data returned by qemu-img info for LUKSv1 based images.
Change-Id: I573396116e10cf87f80f1ded55f2cd8f498859e4
When the resource tracker has to lock a compute host for updates or
inspection, it uses a single semaphore. In most cases, this is fine, as
a compute process only is tracking one hypervisor. However, in Ironic, it's
possible for one compute process to track many hypervisors. In this
case, wait queues for instance claims can get "stuck" briefly behind
longer processing loops such as the update_resources periodic job. The
reason this is possible is because the oslo.lockutils synchronized
library does not use fair locks by default. When a lock is released, one
of the threads waiting for the lock is randomly allowed to take the lock
next. A fair lock ensures that the thread that next requested the lock
will be allowed to take it.
This should ensure that instance claim requests do not have a chance of
losing the lock contest, which should ensure that instance build
requests do not queue unnecessarily behind long-running tasks.
This includes bumping the oslo.concurrency dependency; fair locks were
added in 3.29.0 (I37577becff4978bf643c65fa9bc2d78d342ea35a).
Change-Id: Ia5e521e0f0c7a78b5ace5de9f343e84d872553f9
Related-Bug: #1864122
This includes a couple of fixes and removes the need for custom
workarounds, like the removal of the 'address_string' helper at [1].
[1] https://github.com/novnc/websockify/commit/be9823bf
Change-Id: I9b1d57fd513386e552afbbc8a59cf3a4c54b6908
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This uses the COMPUTE_SAME_HOST_COLD_MIGRATE trait in the API during a
cold migration to filter out hosts that cannot support same-host cold
migration, which is all of them except for the hosts using the vCenter
driver.
For any nodes that do not report the trait, we won't know if they don't
because they don't support it or if they are not new enough to report
it, so the API has a service version check and will fallback to old
behavior using the config if the node is old. That compat code can be
removed in the next release.
As a result of this the FakeDriver capabilities are updated so the
FakeDriver no longer supports same-host cold migration and a new fake
driver is added to support that scenario for any tests that need it.
Change-Id: I7a4b951f3ab324c666ab924e6003d24cc8e539f5
Closes-Bug: #1748697
Related-Bug: #1811235
This reverts commit 33a2a5f2e4. We're now
at M1 and the DevStack default has been changed. We no longer need to
pretend to support Python 2.7. This is slightly modified to drop an
additional Python 2.7-only dependency from lower-constraints.txt.
[1] I52b03caee0ba700da3a15035201ea6cd91baa06b
Change-Id: I1068850072e639ee268d68ca82249f569a40310e
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Revert part of the changes from
Ie1a0cbd82a617dbcc15729647218ac3e9cd0e5a9
This way it is possible again for nova to be installed
using python 2, thus allowing other devstack-based
jobs to work until everything switches to python 3
according the plans.
While py2 support may break anytime, its days are limited anyway.
Change-Id: I99ff84e6c1c033c28d499b0b8ac17ac5b0565f61
It's Ussuri. We can *finally* stop testing Python 2 [1]. Time to party.
We don't attempt any cleanup but simply stop testing with Python 2,
indicate that we only support Python 3 via 'setup.cfg' and remove any
Python 2 only dependencies. Our 'tox.ini' is modified such that
'functional' now runs with 'python3', whatever that may point to, though
the gate will only use a versioned variant (currently
'functional-py36').
This should free up a significant amount of resources from the gate and
let us start using Python 3 idioms in our code. Win-win.
[1] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html#python2-deprecation-timeline
Change-Id: Ie1a0cbd82a617dbcc15729647218ac3e9cd0e5a9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
There is a bug in oslo messaging fake RPC driver bug 1529084. This
driver is used in the nova functional test and the bug currently worked
around in the nova production code.
As the bug 1529084 is fixed now in oslo.messaging 10.3.0 we can remove
the workaround in the nova code by bumping the minimum version of
oslo.messaging.
Change-Id: I4a32a688c7ceb05c263a0e93a91fb9b8ff0c65d4
Related-Bug: #1529084
In nova.policy.authorize() method, it convert the context
object to policy value by context.to_policy_values() and then
pass that into oslo.policy enforcer authorize() method.
This is fine till now and does not cause any issue but when
scope_type is set on policy and context then scope info is
not passed correctly.
In case of system scope, oslo.policy check for a key called 'system'
in creds. The oslo.context library uses `system_scope` instead[1],
and the compatibility between both oslo.policy and oslo.context
are handled when complete context is passed into oslo_policy[2].
If nova convert the context object to policy values then system scope
info is not passed into the oslo_policy. Better way is to pass the complete
context object to oslo_policy and let oslo_policy fetch the system scope
info in correct way.
Update the lower constraints for oslo.policy and oslo.context to have
system scope checks feature.
Partial implement blueprint policy-defaults-refresh
[1] https://github.com/openstack/oslo.context/blob/f65408df5cd5924f2879c3ee94d07fd27cb2cf73/oslo_context/context.py#L321
[2] https://github.com/openstack/oslo.policy/blob/b9fd10e2612f26c93d49c168a0408aba6d20e5bf/oslo_policy/policy.py#L994
Change-Id: I847fc44e62065e3d26e5595e178b83912ab5d19b
We have at least one use case [1] for identifying resource providers
which represent compute nodes. There are a few ways we could do that
hackishly (e.g. [2], [3]) but the clean way is to have nova-compute mark
the provider with a trait, since nova-compute knows which one it is
anyway.
This commit uses the COMPUTE_NODE trait for this purpose, and bumps the
os-traits requirement to 1.1.0 where it is introduced.
Arguably this is a no-op until something starts using it, but a release
note is added anyway warning that all compute nodes should be upgraded
to ussuri (or the trait added manually) for the trait to be useful.
[1] https://review.opendev.org/#/c/670112/7/nova/cmd/manage.py@2921
[2] Assume a provider with a certain resource class, like MEMORY_MB, is
always a compute node. This is not necessarily future-proof (maybe all
MEMORY_MB will someday reside on NUMA node providers; similar for other
resource classes) and isn't necessarily true in all cases today anyway
(ironic nodes don't have MEMORY_MB inventory) and there's also currently
no easy way to query for that (GET /resource_providers?MEMORY_MB:1 won't
return "full" providers, and you can't ask for :0).
[3] Assume a root provider without the MISC_SHARES_VIA_AGGREGATE trait
is a compute node. This assumes you're only using placement for nova-ish
things.
Change-Id: I4cb9cbe1e02c3f6c6148f73a38d10e8db7e61b1a
Cinder introduced "shared_targets" and "service_uuid" fields in volumes
to allow volume consumers to protect themselves from unintended leftover
devices when handling iSCSI connections with shared targets.
Nova avoids races caused by automatic rescans on iSCSI volumes when
detaching a volume while Cinder is mapping another volume to the same
host by locking and only allowing one attach or one detach operation for
each server to happen at a given time if "shared_targets" is set to
True.
When using an up to date Open iSCSI initiator we don't need to use
locks, as it is possible to disable automatic LUN scans (which are the
real cause of the leftover devices), and OS-Brick already supports this
feature.
Currently Nova is blindly locking whenever "shared_targets" is set to
True, even when the iSCSI initiator and OS-Brick are already presenting
such races, which introduces unnecessary locking and serialization on
the connection of volumes.
This patch uses the new context manager introduced in OS-Brick to allow
Nova to abstract its code from all this storage internal details and to
only lock when it's really necessary.
Depends-On: I4970363301d5d1f4e7d0f07e09b34d15ee6884c3
Closes-Bug: #1800515
Change-Id: Ie9106d5832d6a728ea97a8dbb5ddb5dcc17a2ec4
The combined fixes for the two related bugs resolve the problem where
SIGHUP breaks the nova-compute service. Bump the minimum requirements
for oslo.privsep and oslo.service to make sure these fixes are in place,
and add a reno to advertise resolution of the issue.
This also bumps oslo.utils to match the lower constraint from
oslo.service.
Change-Id: I39ead744b21a4423352a88573f327273e4d09630
Related-Bug: #1794708
Related-Bug: #1715374
When instance_get_all_uuids_by_hosts was added [1] some follow up
cleanups where suggested. This change provides them:
* removal of redundance in docstring
* moving docstring to the public method, rather than the private
implementation
* more clarity on the type of the default (defaultdict(list)) and
the implications thereof
* Using an sa.bindparam in the 'in_' call. This requires that the
SQLAlchemy requirment be raised to at least 1.2.0 where the feature
was added. 1.2.19, the latest bugfix release, is chosen.
[1] If92fe8b75d20a738f37e2a74c52c59bfc699a74f
Change-Id: Ib538ab070d73b06ddeb9fea3af149304e40952ec
Add a new "hw:mem_encryption" extra spec parameter, and a new
"hw_mem_encryption" image property, which indicate that any guest
booted with that extra spec parameter or image property respectively
needs to be booted with its memory hardware-encrypted.
This is achieved by converting the requirement stated in the extra
spec parameter and/or image property into an additional extra spec
parameter which requests resources for one slot of the inventory of
the new MEM_ENCRYPTION_CONTEXT resource class (introduced in
os-resource-classes 0.4.0). The inventory will be provided by the
follow-up commit I659cb77f12a38a4d2fb118530ebb9de88d2ed30d.
Since future commits adding support for SEV to guest XML config will
also need to know at launch-time whether memory encryption has been
requested, add a reusable mem_encryption_requested() function to the
nova.virt.hardware library for detecting which of the extra spec /
image property (if either) have requested encrypted memory.
If both the extra spec parameter and the image property are explicitly
specified and they contradict each other, or if either request memory
encryption but the image does not have hw_firmware_type set to UEFI,
then log an error and raise a new generic FlavorImageConflict
exception. This exception can also be useful in the future for
handling other similar conflicts. In this particular use case,
FlavorImageConflict is raised by mem_encryption_requested(), and then
if caught during API call validation, it's re-raised as
HTTPBadRequest.
In order to test this code, we need to construct various ImageMeta
objects containing fake data and a ImageMetaProps instance for each.
This is a slightly fiddly task which future patches in the SEV series
will also need to perform, so add a helper to nova.tests.unit.image.fake
for this.
blueprint: amd-sev-libvirt-support
Change-Id: I8c63b5cc5ad97ce831adb2eb96a995ebc798ecb7
Lee Yarwood