During development of a new git commit, locally running a whole unit
or functional test suite to check every minor code change is
prohibitively expensive. For maximum developer productivity and
happiness, it's generally desirable to make the feedback loop of the
traditional red/green cycle as quick as possible.
So add run-tests-for-diff.sh and run-tests.py to the tools/
subdirectory, using a few tricks as explained below to help with this.
run-tests.py takes a list of files on STDIN, filters the list for
tests which can be run in the current tox virtualenv, and then runs
them with the correct stestr options.
run-tests-for-diff.sh is a simple wrapper around run-tests.py which
determines which tests to run using output from "git diff". This
allows running only the test files changed/added in the working tree:
tools/run-tests-for-diff.sh
or by a single commit:
tools/run-tests-for-diff.sh mybranch^!
or a range of commits, e.g. a branch containing a whole patch series
for a blueprint:
tools/run-tests-for-diff.sh gerrit/master..bp/my-blueprint
It supports the same "-HEAD" invocation syntax as flake8wrap.sh (as
used by the "fast8" tox environment):
tools/run-tests-for-diff.sh -HEAD
run-tests.py uses two tricks to make test runs as quick as possible:
1. It's (already) possible to speed up running of tests by
source'ing the "activate" file for the desired tox virtualenv,
e.g.
source .tox/py36/bin/activate
and then running stestr directly. This saves a few seconds by
skipping the overhead introduced by running tox.
2. When only one test file needs to be run, specifying the -n option
to stestr will skip the costly test discovery phase, saving
several more valuable seconds.
Future commits could build on top of this work, harnessing a framework
such as watchdog / watchmedo[0] or Guard[1] in order to automatically
run relevant tests every time your editor saves changes to a .py file.
[0] https://github.com/gorakhargosh/watchdog - Python-based
[1] https://guardgem.org - probably best in class, but Ruby-based so
maybe unacceptable for use within Nova.
Change-Id: I9a9bda5d29bbb8d8d77f769cd1abf7c42a18c36b
Since If32bca070185937ef83f689b7163d965a89ec10a some of the service
update api tests are slow because setting a compute enabled or
disabled makes an RPC call to the given compute service. These tests
does not have the proper RPC setup so that call times out. The compute
api only logs an error for the timeout so the tests wasn't failing.
This patch stubs the RPC call as that is not necessary for the
api testing and therefore speeds up the tests.
Change-Id: If0960de896a67fd6bfca230b8915a45cb7af99b7
While addressing a nit from change
I8e5a122cc547222249973cf595d90c2d8d5658d4, I realized pretty much every
test using the nova.tests.functional.libvirt.base.ServersTestBase' base
class was now mocking the same things. This results in a lot of noise
and distracts from what the tests are trying to do. Centralize the
common mocks in the base class and clean up said noise.
Change-Id: I5895865751e8e1fb08b3515bc9f8119cfcb9f35e
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
There is no method called_once_with() in Mock object.
Use assert_called_once_with() or assert_has_calls() instead.
Change-Id: I9f73fcbe7c3dfd64e75ac8224c13934b03443cd5
Closes-Bug: #1544522
test_driver.LibvirtConnTestCase.test_cpu_info() and
test_driver.LibvirtConnTestCase.test_get_instance_capabilities() were
incorrectly adding LibvirtConfigGuest fixture objects to the guests
attribute of LibvirtConfigCaps, rather than LibvirtConfigCapsGuest
objects which is what the parse_dom() method of LibvirtConfigCaps
does. LibvirtConfigCaps is intended to capture all useful information
returned by libvirt's getCapabilities API call, and
LibvirtConfigCapsGuest is intended to capture a small subset of that.
In contrast, LibvirtConfigGuest is intended for holding the entire
configuration of a libvirt guest domain.
So fix this to instead add LibvirtConfigCapsGuest fixtures.
Note that despite this bug, the tests previously passed through
duck-typing because LibvirtConfigCaps does not define __slots__ or any
other mechanism for limiting how attributes can be set on instances.
This is required by another commit in the SEV series in the near
future which will alter the fields within LibvirtConfigCapsGuest.
blueprint: amd-sev-libvirt-support
Change-Id: Ic4cbbaf81938fc88980e44c4a17fe388f9c2b92b
matchers.XMLMatches supports three options:
- allow_mixed_nodes
- skip_empty_text_nodes
- skip_values
However these are not accessible when using the matcher via
assertXmlEqual(). So extend the latter with an *options
dictionary to allow options to be passed through.
This will be used by another commit in the SEV series in the near
future which needs the allow_mixed_nodes option to be enabled to
compare XML fragments with elements in different orders.
blueprint: amd-sev-libvirt-support
Change-Id: I31c60771891c3a2eb06fbc780ba2d369465498ec
This patch adds a new external event called "power-update"
through which ironic will convey all (power_off and power_on)
power state changes (running -> shutdown or shutdown -> running
will be the only ones handled by nova and the rest will be ignored)
on a physical instance to nova. The database will be updated
accordingly to reflect the real vm_state and power_state of the
instance. This way nova will not be able to enforce
an incorrect power state on the physical instance during
the periodic "sync_power_states" task.
Implements blueprint nova-support-instance-power-update
Story: 2004969
Task: 29423
Change-Id: I2b292050cc3ce5ef625659f5a1fe56bb76072496
This patch removes the legacy code for image checksumming
as well as configuration values that are not longer being
used.
Change-Id: I9c552e33456bb862688beaabe69f2b72bb8ebcce
As we work out the kinks in openstacksdk to smooth the road for nova's
usage thereof, and until openstacksdk can provide opaque fixtures that
it can keep up to date as its internals change, we've been needing to
stub things out incrementally.
This commit more aggressively mocks the piece of openstacksdk that
keeps changing under us. It effectively does what [1] does, but without
the get_endpoint() sanity check, which blows up in unit/functional test
because it tries to do real ksa auth.
[1] https://review.opendev.org/#/c/675135/
Change-Id: I1ef2ca51da2dc25026a2fbdcfb1ed5199aaa518b
Fix collapse of 'host_status' description in the follwoing APIs
in the compute API reference.
- PUT /servers/{server_id}
- POST /servers/{server_id}/action (rebuild)
Change-Id: I003f9a81ac6f7e0ec13a24db3fda1b7ff6612bc5
Closes-Bug: #1840094
_detect_nbd_devices uses the filter
builtin internally to filter valid devices.
In python 2, filter returns a list. In python 3,
filter returns an iterable or generator function.
This change eagerly converts the result of calling filter
to a list to preserve the python 2 behaviour under python 3.
Closes-Bug: #1840068
Change-Id: I25616c5761ea625a15d725777ae58175651558f8
This adds the code to CrossCellMigrationTask which
executes the TargetDBSetupTask sub-task and stores
some fields on the main task for use later when
dealing with the target cell.
Part of blueprint cross-cell-resize
Change-Id: I00683acee216c0c0ad87be3eb5ec832f20f054c7
This adds the main conductor task that will orchestrate
a cross-cell resize. This will get called from the
existing conductor MigrationTask when that task determines
the scheduler picked a target host in another cell for the
resize operation.
The only thing CrossCellMigrationTask does in this change
is perform checks on the neutron and cinder APIs to make
sure they are new enough for what we need before we go any
further with the resize.
Part of blueprint cross-cell-resize
Change-Id: I4d181b44494f3b0b04537d5798537831c8fdf400
The test_migrate_disk_and_power_off_crash_finish_revert_migration test
needs to simulate a compute host crash at a certain point. It stops the
execution at a certain point by injecting a sleep then simulating a
compute restart. However the sleep is just 30 seconds which allows the
stopped function to return while other functional tests are running in
the same test worker process making those tests fail in a weird way.
One simple solution is to add a big enough sleep to the test that will
never return before the whole functional test execution. This patch
proposes a million seconds which is more than 277 hours. Similar to how
the other test in this test package works. This solution is hacky but
simple. A better solution would be to further enhance the capabilities
of the functional test env supporting nova-compute service crash / kill
+ restart.
Change-Id: Ib0d142806804e9113dd61d3a7ec15a98232775c8
Closes-Bug: #1839515
The nova.objects.base.obj_equal_prims returns True or False.
It does not assert anything.
So the return value should be asserted in tests.
Add assertTrue where the nova.objects.base.obj_equal_prims is called.
Change-Id: I49460ec3b572ee14b32229e771a5499ff91e8722
Closes-Bug: #1839853
There is a unique index on the compute_nodes.uuid column which
means we can't have more than one compute_nodes record in the
same DB with the same UUID even if one is soft deleted because
the deleted column is not part of that unique index constraint.
This is a problem with ironic nodes where the node is 1:1 with
the compute node record, and when a node is undergoing maintenance
the driver doesn't return it from get_available_nodes() so the
ComputeManager.update_available_resource periodic task (soft)
deletes the compute node record, but when the node is no longer
under maintenance in ironic and the driver reports it, the
ResourceTracker._init_compute_node code will fail to create the
ComputeNode record again because of the duplicate uuid.
This change handles the DBDuplicateEntry error in compute_node_create
by finding the soft-deleted compute node with the same uuid and
simply updating it to no longer be (soft) deleted.
Closes-Bug: #1839560
Change-Id: Iafba419fe86446ffe636721f523fb619f8f787b3
This adds a functional test which recreates bug 1839560
where the driver reports a node, then no longer reports
it so the compute manager deletes it, and then the driver
reports it again later (this can be common with ironic
nodes as they undergo maintenance). The issue is that since
Ia69fabce8e7fd7de101e291fe133c6f5f5f7056a in Rocky, the
ironic node uuid is re-used for the compute node uuid but
there is a unique constraint on the compute node uuid so
when trying to create the compute node once the ironic node
is available again, the compute node create fails with a
duplicate entry error due to the duplicate uuid. To recreate
this in the functional test, a new fake virt driver is added
which provides a predictable uuid per node like the ironic
driver. The test also shows that archiving the database is
a way to workaround the bug until it's properly fixed.
Change-Id: If822509e906d5094f13a8700b2b9ed3c40580431
Related-Bug: #1839560
If ComputeNode.create() fails, the update_available_resource
periodic will not try to create it again because it will be
mapped in the compute_nodes dict and _init_compute_node will
return early but trying to save changes to that ComputeNode
object later will fail because there is no id on the object,
since we failed to create it in the DB.
This simply reverses the logic such that we only map the
compute node if we successfully created it.
Some existing _init_compute_node testing had to be changed
since it relied on the order of when the ComputeNode object
is created and put into the compute_nodes dict in order
to pass the object along to some much lower-level PCI
tracker code, which was arguably mocking too deep for a unit
test. That is changed to avoid the low-level mocking and
assertions and just assert that _setup_pci_tracker is called
as expected.
Change-Id: I9fa1d509a3de405d6246fb8670612c65c10cc93b
Closes-Bug: #1839674
Since If32bca070185937ef83f689b7163d965a89ec10a some of the service
update notification tests are slow because setting a compute enabled or
disabled makes an RPC call to the given compute service. These tests
does not have the proper RPC setup so that call times out. Since the
notification is still generated the test passes. Waiting for this
timeout steals 16 minutes of test execution time unnecessary.
This patch stubs the RPC call as that is not necessary for the
notification testing and therefore speeds up the tests.
Change-Id: I0269c4bd3da0e58593811b11985deb237eefe59e
This microversion implements below API cleanups:
1. 400 for unknown param for query param and for request body.
2. Making server representation always consistent among all APIs
returning the complete server representation.
3. Change the default return value of ``swap`` field from the empty string
to 0 (integer) in flavor APIs.
4. Return ``servers`` field always in the response of GET
hypervisors API even there are no servers on hypervisor
Details: https://specs.openstack.org/openstack/nova-specs/specs/train/approved/api-consistency-cleanup.html
Partial-Implements: blueprint api-consistency-cleanup
Change-Id: I9d257a003d315b84b937dcef91f3cb41f3e24b53
Adam Spiers