Default value of 'CONF.oslo_policy.policy_file' config option
has been changed from 'policy.json' to 'policy.yaml'. If new default
file 'policy.yaml' does not exist but old default 'policy.json' exist
then fallback to use old default file.
An upgrade checks is added to check the policy_file format and
fail upgrade checks if it is JSON formatted.
Added a warning in policy doc about JSON formatted file is deprecated,
also removed all the reference to policy.json file in doc as well as
in tests.
Related Blueprint: policy-json-to-yaml
Closes-Bug: #1875418
Change-Id: Ic4d3b998bb9701cb1e3ef12d9bb6f4d91cc19c18
Several version specified in our lower-constraints files had conflicting
dependencies. This updates a few packages to avoid those conflicts. It
also removes the linters that are tracked in the global requirements
blocklist since we do not need to enforce lower-constraints for linters.
Change-Id: Iae93f0631d77794f093fd330e849d2fd805b238f
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This series implements the referenced blueprint to allow for specifying
custom resource provider traits and inventories via yaml config files.
This first commit includes schema version 1.0 and functions to load and
validate against that schema.
Since this patch, package ddt>=1.2.1 is required to run tests.
Co-Author: Tony Su <tao.su@intel.com>
Author: Dustin Cowles <dustin.cowles@intel.com>
Blueprint: provider-config-file
Change-Id: I58099726f799c427e9174a0dcce9889344d51e7c
To support new AVX512 cpu feature, this patch adds the mapping
between cpu features reported by libvirt [1] and corresponding
os-traits [2]. The traits include 'AVX512VBMI', 'AVX512IFMA',
'AVX512VBMI2', 'AVX512BITALG', 'AVX512VAES', 'AVX512GFNI',
'AVX512VPCLMULQDQ' and 'AVX512VPOPCNTDQ'.
[1] https://github.com/libvirt/libvirt/blob/master/src/cpu_map/x86_features.xml
[2] https://review.opendev.org/#/c/728576/
Depends-On: I58cba02f204d2e19a1a838852c628694edd47d73
Change-Id: I212a84e5cae15bc133aafd538d55f85149ab360f
oslo has new flag to disable the warning for default check_str
change and keep logging warning if policy name is changed or it is
marked for removal.
Policy defaults refresh BP work changed the defaults for all the
policy so logging their warning is dangerousa and fill up the log file.
Once we switch to new defaults then we can enable them if needed.
Logging warning for name change and marked for removal make sense
when old rules are override by operator so the new flag does not control
those. For policy defaults refresh BP work, we do not have many policies
in this category so logging their warning is ok.
This commits reset the testing flag 'suppress_deprecation_warnings'
default value back to False. Test policy fixture sets that to True to
avoid any warnig during tests run.
Bumping the oslo policy version where this new flag is present
- https://review.opendev.org/#/c/719547/
Partial implement blueprint policy-defaults-refresh
Change-Id: Iaa6baf6877890babb6205bdb3ee2e1a2b28ebd9c
Add the validation framework necessary to verify extra specs along with
the definitions for every extra spec we currently recognize in-tree.
None of this is currently used since we don't have the API microversions
wired up, but that will come in a future patch.
Note that we must add the H238 hacking check to the ignore list here,
since this includes our first use of Python 3-type classes without the
explicit 'object' subclass. This can be removed when that check is
removed from hacking.
Part of blueprint flavor-extra-spec-validators
Change-Id: Ib64a1348cce1dca995746214616c4f33d9d664bd
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Building on the ``[workarounds]/disable_native_luksv1``
configurable introduced in Ia500eb614cf575ab846f64f4b69c9068274c8c1f
this change introduces another workaround configurable that when enabled
will connect RBD volumes to the compute host as block devices using
os-brick.
When used togther both options allow operators to workaround recently
discovered performance issues in the libgcrypt library used by QEMU when
natively decrypting LUKSv1 encrypted disks.
For now the extend_volume method raises a NotImplemented error in-line
with the underlying method in os-brick. Future work will be required to
both support this in os-brick and wire up the required calls in the
volume driver.
This workaround is temporary and will be removed during the W release
once all impacted distributions have been able to update their versions
of the libgcrypt library.
Finally os-brick 3.0.1 is now required as it provides the
Id507109df80391699074773f4787f74507c4b882 fix when attempting to
diconnect locally attached RBD volumes.
Closes-Bug: #1869184
Change-Id: Ied3732042738a6194b635c55e0304d71a6fb66e3
This change is a follow up to I0c3f14100a18107f7e416293f3d4fcc641ce5e55
and removes the direct call to nova.privsep.qemu with one to the images
API that now returns an oslo_utils.imageutils.QemuImgInfo object.
Version 4.1.0 of oslo.utils introducing support for the format-specific
data returned by qemu-img info for LUKSv1 based images.
Change-Id: I573396116e10cf87f80f1ded55f2cd8f498859e4
When the resource tracker has to lock a compute host for updates or
inspection, it uses a single semaphore. In most cases, this is fine, as
a compute process only is tracking one hypervisor. However, in Ironic, it's
possible for one compute process to track many hypervisors. In this
case, wait queues for instance claims can get "stuck" briefly behind
longer processing loops such as the update_resources periodic job. The
reason this is possible is because the oslo.lockutils synchronized
library does not use fair locks by default. When a lock is released, one
of the threads waiting for the lock is randomly allowed to take the lock
next. A fair lock ensures that the thread that next requested the lock
will be allowed to take it.
This should ensure that instance claim requests do not have a chance of
losing the lock contest, which should ensure that instance build
requests do not queue unnecessarily behind long-running tasks.
This includes bumping the oslo.concurrency dependency; fair locks were
added in 3.29.0 (I37577becff4978bf643c65fa9bc2d78d342ea35a).
Change-Id: Ia5e521e0f0c7a78b5ace5de9f343e84d872553f9
Related-Bug: #1864122
This includes a couple of fixes and removes the need for custom
workarounds, like the removal of the 'address_string' helper at [1].
[1] https://github.com/novnc/websockify/commit/be9823bf
Change-Id: I9b1d57fd513386e552afbbc8a59cf3a4c54b6908
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This uses the COMPUTE_SAME_HOST_COLD_MIGRATE trait in the API during a
cold migration to filter out hosts that cannot support same-host cold
migration, which is all of them except for the hosts using the vCenter
driver.
For any nodes that do not report the trait, we won't know if they don't
because they don't support it or if they are not new enough to report
it, so the API has a service version check and will fallback to old
behavior using the config if the node is old. That compat code can be
removed in the next release.
As a result of this the FakeDriver capabilities are updated so the
FakeDriver no longer supports same-host cold migration and a new fake
driver is added to support that scenario for any tests that need it.
Change-Id: I7a4b951f3ab324c666ab924e6003d24cc8e539f5
Closes-Bug: #1748697
Related-Bug: #1811235
This reverts commit 33a2a5f2e4. We're now
at M1 and the DevStack default has been changed. We no longer need to
pretend to support Python 2.7. This is slightly modified to drop an
additional Python 2.7-only dependency from lower-constraints.txt.
[1] I52b03caee0ba700da3a15035201ea6cd91baa06b
Change-Id: I1068850072e639ee268d68ca82249f569a40310e
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Revert part of the changes from
Ie1a0cbd82a617dbcc15729647218ac3e9cd0e5a9
This way it is possible again for nova to be installed
using python 2, thus allowing other devstack-based
jobs to work until everything switches to python 3
according the plans.
While py2 support may break anytime, its days are limited anyway.
Change-Id: I99ff84e6c1c033c28d499b0b8ac17ac5b0565f61
It's Ussuri. We can *finally* stop testing Python 2 [1]. Time to party.
We don't attempt any cleanup but simply stop testing with Python 2,
indicate that we only support Python 3 via 'setup.cfg' and remove any
Python 2 only dependencies. Our 'tox.ini' is modified such that
'functional' now runs with 'python3', whatever that may point to, though
the gate will only use a versioned variant (currently
'functional-py36').
This should free up a significant amount of resources from the gate and
let us start using Python 3 idioms in our code. Win-win.
[1] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html#python2-deprecation-timeline
Change-Id: Ie1a0cbd82a617dbcc15729647218ac3e9cd0e5a9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
There is a bug in oslo messaging fake RPC driver bug 1529084. This
driver is used in the nova functional test and the bug currently worked
around in the nova production code.
As the bug 1529084 is fixed now in oslo.messaging 10.3.0 we can remove
the workaround in the nova code by bumping the minimum version of
oslo.messaging.
Change-Id: I4a32a688c7ceb05c263a0e93a91fb9b8ff0c65d4
Related-Bug: #1529084
In nova.policy.authorize() method, it convert the context
object to policy value by context.to_policy_values() and then
pass that into oslo.policy enforcer authorize() method.
This is fine till now and does not cause any issue but when
scope_type is set on policy and context then scope info is
not passed correctly.
In case of system scope, oslo.policy check for a key called 'system'
in creds. The oslo.context library uses `system_scope` instead[1],
and the compatibility between both oslo.policy and oslo.context
are handled when complete context is passed into oslo_policy[2].
If nova convert the context object to policy values then system scope
info is not passed into the oslo_policy. Better way is to pass the complete
context object to oslo_policy and let oslo_policy fetch the system scope
info in correct way.
Update the lower constraints for oslo.policy and oslo.context to have
system scope checks feature.
Partial implement blueprint policy-defaults-refresh
[1] https://github.com/openstack/oslo.context/blob/f65408df5cd5924f2879c3ee94d07fd27cb2cf73/oslo_context/context.py#L321
[2] https://github.com/openstack/oslo.policy/blob/b9fd10e2612f26c93d49c168a0408aba6d20e5bf/oslo_policy/policy.py#L994
Change-Id: I847fc44e62065e3d26e5595e178b83912ab5d19b
We have at least one use case [1] for identifying resource providers
which represent compute nodes. There are a few ways we could do that
hackishly (e.g. [2], [3]) but the clean way is to have nova-compute mark
the provider with a trait, since nova-compute knows which one it is
anyway.
This commit uses the COMPUTE_NODE trait for this purpose, and bumps the
os-traits requirement to 1.1.0 where it is introduced.
Arguably this is a no-op until something starts using it, but a release
note is added anyway warning that all compute nodes should be upgraded
to ussuri (or the trait added manually) for the trait to be useful.
[1] https://review.opendev.org/#/c/670112/7/nova/cmd/manage.py@2921
[2] Assume a provider with a certain resource class, like MEMORY_MB, is
always a compute node. This is not necessarily future-proof (maybe all
MEMORY_MB will someday reside on NUMA node providers; similar for other
resource classes) and isn't necessarily true in all cases today anyway
(ironic nodes don't have MEMORY_MB inventory) and there's also currently
no easy way to query for that (GET /resource_providers?MEMORY_MB:1 won't
return "full" providers, and you can't ask for :0).
[3] Assume a root provider without the MISC_SHARES_VIA_AGGREGATE trait
is a compute node. This assumes you're only using placement for nova-ish
things.
Change-Id: I4cb9cbe1e02c3f6c6148f73a38d10e8db7e61b1a
Cinder introduced "shared_targets" and "service_uuid" fields in volumes
to allow volume consumers to protect themselves from unintended leftover
devices when handling iSCSI connections with shared targets.
Nova avoids races caused by automatic rescans on iSCSI volumes when
detaching a volume while Cinder is mapping another volume to the same
host by locking and only allowing one attach or one detach operation for
each server to happen at a given time if "shared_targets" is set to
True.
When using an up to date Open iSCSI initiator we don't need to use
locks, as it is possible to disable automatic LUN scans (which are the
real cause of the leftover devices), and OS-Brick already supports this
feature.
Currently Nova is blindly locking whenever "shared_targets" is set to
True, even when the iSCSI initiator and OS-Brick are already presenting
such races, which introduces unnecessary locking and serialization on
the connection of volumes.
This patch uses the new context manager introduced in OS-Brick to allow
Nova to abstract its code from all this storage internal details and to
only lock when it's really necessary.
Depends-On: I4970363301d5d1f4e7d0f07e09b34d15ee6884c3
Closes-Bug: #1800515
Change-Id: Ie9106d5832d6a728ea97a8dbb5ddb5dcc17a2ec4
The combined fixes for the two related bugs resolve the problem where
SIGHUP breaks the nova-compute service. Bump the minimum requirements
for oslo.privsep and oslo.service to make sure these fixes are in place,
and add a reno to advertise resolution of the issue.
This also bumps oslo.utils to match the lower constraint from
oslo.service.
Change-Id: I39ead744b21a4423352a88573f327273e4d09630
Related-Bug: #1794708
Related-Bug: #1715374
When instance_get_all_uuids_by_hosts was added [1] some follow up
cleanups where suggested. This change provides them:
* removal of redundance in docstring
* moving docstring to the public method, rather than the private
implementation
* more clarity on the type of the default (defaultdict(list)) and
the implications thereof
* Using an sa.bindparam in the 'in_' call. This requires that the
SQLAlchemy requirment be raised to at least 1.2.0 where the feature
was added. 1.2.19, the latest bugfix release, is chosen.
[1] If92fe8b75d20a738f37e2a74c52c59bfc699a74f
Change-Id: Ib538ab070d73b06ddeb9fea3af149304e40952ec
Add a new "hw:mem_encryption" extra spec parameter, and a new
"hw_mem_encryption" image property, which indicate that any guest
booted with that extra spec parameter or image property respectively
needs to be booted with its memory hardware-encrypted.
This is achieved by converting the requirement stated in the extra
spec parameter and/or image property into an additional extra spec
parameter which requests resources for one slot of the inventory of
the new MEM_ENCRYPTION_CONTEXT resource class (introduced in
os-resource-classes 0.4.0). The inventory will be provided by the
follow-up commit I659cb77f12a38a4d2fb118530ebb9de88d2ed30d.
Since future commits adding support for SEV to guest XML config will
also need to know at launch-time whether memory encryption has been
requested, add a reusable mem_encryption_requested() function to the
nova.virt.hardware library for detecting which of the extra spec /
image property (if either) have requested encrypted memory.
If both the extra spec parameter and the image property are explicitly
specified and they contradict each other, or if either request memory
encryption but the image does not have hw_firmware_type set to UEFI,
then log an error and raise a new generic FlavorImageConflict
exception. This exception can also be useful in the future for
handling other similar conflicts. In this particular use case,
FlavorImageConflict is raised by mem_encryption_requested(), and then
if caught during API call validation, it's re-raised as
HTTPBadRequest.
In order to test this code, we need to construct various ImageMeta
objects containing fake data and a ImageMetaProps instance for each.
This is a slightly fiddly task which future patches in the SEV series
will also need to perform, so add a helper to nova.tests.unit.image.fake
for this.
blueprint: amd-sev-libvirt-support
Change-Id: I8c63b5cc5ad97ce831adb2eb96a995ebc798ecb7
In version 0.35.0, openstacksdk added a strict_proxies kwarg to the
Connection constructor [1].
Without it, openstacksdk tries really hard to give us an Adapter, which
in the case of the service being down can mean we default to the catalog
endpoint without doing any discovery. This should usually work; but may
break in cases where the discovery document (at the catalog endpoint)
points to different URLs for versioned endpoints.
This commit adds a check_service bool kwarg to get_sdk_adapter which, if
True, uses strict_proxies to create the Connection, and causing
get_sdk_adapter to raise a ServiceUnavailable exception if the service
is down.
This can be used for services like Ironic, where we're set up to
tolerate connect failures on startup. But it should not be used for
services like Placement, where we expect getting the adapter to succeed,
and are instead tolerant of failures making the actual API calls.
[1] https://review.opendev.org/#/c/676837/
This dependency bumps the openstacksdk u-c in the requirements project.
Depends-On: https://review.opendev.org/678207
Change-Id: I86e038af8a96e113a754b2fdb3698acd3783c1c8
A number of different efforts are going to need to make use of
openstacksdk 0.34.0 and keystoneauth1 3.16.0, so rather than bump the
minimum in all of those, bump it in one place.
Also, this gives us the opportunity to independently validate some of
the fixes (particularly in logging) we were expecting to affect nova.
Change-Id: I87d1dcd299f6547d5f3c3d77e219bf71aba1cff2
'AVX512-VNNI' is the instruction set for vector neural network
instructions supported since CascadeLake CPU. Enabling this
CPU feature in Nova.
Requires 'os-traints' to be greater than '0.16.0'.
Depends-On: Ia421ed500fbc15bf0088a8436ddeb5d8d1196256
Change-Id: I4ee821cba7cd23f0db9dc2c2c83c78ef5e70ad7b
Enables the use of the sdk instead of ksa adapter or python-*client.
It is provided by a get_sdk_adapter method which constructs an
authenticated SDK Connection object using provided service configuration.
This change should be transparent to operators of services which already
use ksa as get_sdk_adapter uses the same conf options from keystoneauth1.
Blueprint: openstacksdk-in-nova
Co-Authored-By: Dustin Cowles <dustin.cowles@intel.com>
Change-Id: I49f364e01e2a18de0c95674654fc72acea019e76
Release 3.15.0 of keystoneauth1 introduced the ability to pass
X-Openstack-Request-Id to request methods (get/put/etc) via a
global_request_id kwarg rather than having to put it in a headers dict.
This commit bumps the minimum ksa level to 3.15.0 and takes advantage of
the new kwarg to replace explicit header construction in
SchedulerReportClient (Placement) and neutronv2/api methods.
Also normalizes the way param lists were being passed from
SchedulerReportClient's REST primitives (get/put/post/delete) into the
Adapter equivalents. There was no reason for them to be different.
Change-Id: I2f6eb50f4cb428179ec788de8b7bd6ef9bbeeaf9
This adds code which hooks in the update_provider_tree flow in the
ResourceTracker, specifically when the RT is generically modifying
the compute-owned traits for the given compute node resource provider.
A future change will add the scheduler request pre-filter and
API code to sync the trait when enabling/disabling a compute service.
This is necessary for two cases specifically:
1. After upgrading an older disabled compute we will sync the trait.
2. If enabling/disabling a compute and the service is down, the API
will not call the compute service to sync the trait. When the
compute service is restarted we will sync the trait on startup
with this code.
The COMPUTE_STATUS_DISABLED trait was added to os-traits in change
Ia8e4487bfb59f764a6817ec8650785ffa902eab5 which is in the 0.15.0
release of os-traits so the requirements are bumped here as well.
Part of blueprint pre-filter-disabled-computes
Change-Id: I3005b46221ac3c0e559e1072131a7e4846c9867c
Version 2.6 of the cryptography library [1] added support for ed25519
ssh keys. This works with OpenSSL >= 1.1.1b.
In nova, we can enable people to use ed25519 ssh keys by using the
necessary cryptography library version. Users must make sure they have
a new enough OpenSSL version, else they won't be able to generate
ed25519 ssh keys using ssh-keygen in the first place. I did a local
test using Ubuntu 18.04 and things "just worked" when I generated a
ed25519 ssh key and imported it into nova. I left a comment on the
launchpad bug accordingly.
This updates our minimum version to the latest available version 2.7.
Closes-Bug: #1555521
[1] https://cryptography.io/en/latest/changelog/#v2-6
Change-Id: Id4a4e1ae4c0acd40c1fc32c3b82a8d8a62d4624d
This release of the Cinder client broke support for the v3
volume-transfer APIs unless microversion 3.55 or higher was requested.
Depends-On https://review.opendev.org/#/c/587877/
Change-Id: Ieb685a476d51d92ad3f153fb3d1fabfb6d5a4376
Ghanshyam Mann