I4f551dc4b57905cab8aa005c5680223ad1b57639 introduced the environment
variable to disable the check-cherry-pick.sh script but forgot to allow
it to be passed into the pep8 tox env.
Change-Id: Ie8a672fd21184c810bfe9c0e3a49582189bf2111
This significantly speeds up our doc build process. This requires a
newer version of 'sphinx-feature-classification' and some tweaks to our
own in-tree extensions. While we're here, we drop the '-d DOCTREE_DIR'
parameter since it's of no use when we blast away our previously built
docs each time we build.
Change-Id: I679da65d44c40880f720df8a2f06286a19eb8d22
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Introduce API microversion 2.88, which makes the following changes to
a number of 'os-hypervisors'. Specifically, the following fields are
dropped from both the '/os-hypervisors/detail' (detailed list) and
'/os-hypervisors/{hypervisor_id}' (show) APIs:
- current_workload
- cpu_info
- vcpus
- vcpus_used
- free_disk_gb
- local_gb
- local_gb_used
- disk_available_least
- free_ram_mb
- memory_mb
- memory_mb_used
- running_vms
In addition, the '/os-hypervisors/statistics' API, which provided a
summary of the above stats but for all hypervisors in the deployment, is
dropped entirely.
Finally, the '/os-hypervisors/{hypervisor}/uptime' API, which provided a
similar response to the '/os-hypervisors/{hypervisor}' API but with an
additional 'uptime' field, has been removed in favour of including this
field in the primary '/os-hypervisors/{hypervisor}' API.
A small tweak to 'tox.ini' that allows us to share some venvs is
included.
Part of blueprint modernize-os-hypervisors-api
Change-Id: I515e484ade6c6455f82a3067940a418a0d7d965a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
In a heavily IO deprived CI VM the db migration tests could take a
significant amount of time and eventually time out. This patch moves the
tests into the same test executor worker process to spread the load
generated by these test in time until a final solution is found. For
example we hope that [1] will help eventually to decrease the load.
[1] https://review.opendev.org/q/topic:bp/compact-db-migrations-wallaby
Change-Id: I6ce930fa86c82da1008089791942b1fff7d04c18
Related-Bug: #1823251
Nova does not run gabbi test since placement is moved to a separate
git repository. So the gabbi related tox.ini comment is removed.
Change-Id: Ic324e3e32fa03478895b32fa583e805ee6c721e2
This patch adds both a tox target and a zuul job to run functional tests
with python3.9
Depends-On: https://review.opendev.org/760932
Change-Id: I672904e9bfb45a66a82331063c7d49c4bc0439df
This linting makes building docs an even more painful process than it
would otherwise be. We already do this as part of the pep8 target,
which the gate runs, so there's no need to do this here.
Change-Id: I36ae872dd21299ad5d165422cb83564eafd89bea
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Add the following hacking rule.
* N366: The assert_has_calls is a method rather than a variable.
Not correct: mock_method.assert_has_calls = [mock.call(0)]
Correct: mock_method.assert_has_calls([mock.call(0)])
This patch is a follow-up patch for
Id094dd90efde09b9a835d4492f4a92b8f8ad296e.
Change-Id: I892f8c23ee44f2b3518776a9705e3543f3115cae
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
These will not be used in a world without the XenAPI driver.
Change-Id: I5bc3c7855b817c4ce2b8919c76be80cceabeec9e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The Python 2.7 Support has been dropped since Ussuri.
So remove hacking rules for compatibility between python 2 and 3.
- [N325] str() and unicode() cannot be used on an exception.
Remove or use six.text_type()
- [N327] Do not use xrange(). xrange() is not compatible with Python 3.
Use range() or six.moves.range() instead.
- [N344] Python 3: do not use dict.iteritems.
- [N345] Python 3: do not use dict.iterkeys.
- [N346] Python 3: do not use dict.itervalues.
See also line 414 in https://etherpad.opendev.org/p/nova-victoria-ptg
Change-Id: If4335b2e8ef5bbabba37598110c1aa8269635c2f
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
This adds a tools/ script that checks any cherry-picked hashes
on the current commit (or a provided commit) to make sure that
all the hashes exist on at least master or stable/.* branches.
This should help avoid accidentally merging stable backports
where one of the hashes along the line has changed due to conflicts.
Change-Id: I4afaa0808b75cc31a8dd14663912c162281a1a42
The N319 check previously asserted that debug-level logs were not
translated. Now that we've removed all log translations, we can
generalize this to all logs. We reuse the same number since these
numbers are really just metadata and not public contracts.
This also allows us to update the N323 and N326 checks, which ensure we
import the translation function, '_', wherever it's used and don't
concatenate translated and non-translated strings. Since we're no longer
translating logs and the '_LE', '_LW' and '_LI' symbols are no longer
provided, we don't need to consider logs in either of these cases.
Change-Id: I64d139ad660bc382e8b9d7c8cd03352b26aadafd
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
mypy is an experimental optional static type checker for Python that
aims to combine the benefits of dynamic (or "duck") typing and static
typing. While still in development, most features are supported and it's
already being used by real world projects like Sphinx. Let's start small
by integrating it into some of the interfaces that nova exposes. We can
further build upon this if it works out.
This change sets up the boilerplate necessary to use mypy in nova. Type
annotations are not included for any module - these will be added
separately. We're calling mypy by way of a script, as this allows us to
store a list of files that we have converted while we're in the process
of adding type annotations where necessary.
Change-Id: I75ab46a6768c4ca2050fdde2b7f8eeb90724c8c6
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
As per Victoria testing runtime[1], we need to tests py3.6,
py3.7, and py3.8.
- py3.7 is being tested with integration jobs.
- py3.6 and py3.8 are tested with unit test jobs.
Nova functional tests are testing py3.6 which can be moved to the latest
python available in this cycle which is 3.8. We do not need to run functional
or integration tests on each supported python version. Testing them on
the latest python version is enough. Coverage of all supported python version
is achieved via unit tests job.
[1] https://governance.openstack.org/tc/reference/runtimes/victoria.html
Change-Id: I1d6a2986fcb0435cfabdd104d202b65329909d2b
nova-tox-functional is py2 job and need placement as required project.
and started failing for incompatible py version:
ERROR: Package 'openstack-placement' requires a different Python: 2.7.17 not in '>=3.6'
- https://zuul.opendev.org/t/openstack/build/b460a8c59ad64e57b871aa3c00638b01/log/job-output.txt#782
Placement master is now py3 only which mean master version will stop
working on py2 env:
- I6f458fb60b5a33b5aa2ce3ab292862ab98eb4670
Current tox env mention the placement master link as deps, where need to use
the stable version for stable jobs. Making deps with version not the
fresh git clone like how other test-requirements.txt deps work.
Change-Id: I96bb9fcb55f2a4fc3aefa01db70b45740db166c8
Add the validation framework necessary to verify extra specs along with
the definitions for every extra spec we currently recognize in-tree.
None of this is currently used since we don't have the API microversions
wired up, but that will come in a future patch.
Note that we must add the H238 hacking check to the ignore list here,
since this includes our first use of Python 3-type classes without the
explicit 'object' subclass. This can be removed when that check is
removed from hacking.
Part of blueprint flavor-extra-spec-validators
Change-Id: Ib64a1348cce1dca995746214616c4f33d9d664bd
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This bumps the version of flake8 and pycodestyle to something much
newer, which resolves a long-standing warning about nested sets and
allows us to use new fangled features like f-strings if we so choose.
Change-Id: I0bb9077f1cea2243b7945e87cfa140f9cf89d558
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
It's Ussuri. We can *finally* stop testing Python 2 [1]. Time to party.
We don't attempt any cleanup but simply stop testing with Python 2,
indicate that we only support Python 3 via 'setup.cfg' and remove any
Python 2 only dependencies. Our 'tox.ini' is modified such that
'functional' now runs with 'python3', whatever that may point to, though
the gate will only use a versioned variant (currently
'functional-py36').
This should free up a significant amount of resources from the gate and
let us start using Python 3 idioms in our code. Win-win.
[1] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html#python2-deprecation-timeline
Change-Id: Ie1a0cbd82a617dbcc15729647218ac3e9cd0e5a9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Change Iba797243d2a137b551223165a1af1a8676bcea02 was a bit
overzealous in using {[testenv]deps} and changed the docs
tox target to also install requirements.txt and
test-requirements.txt which means for docs builds we're
installing things like psycopg2 which we shouldn't be doing
if we don't have the correct native packages installed to
build those types of dependencies.
Change-Id: Ib718911596b93ec6ec7e899210300d2f0d9572ed
Closes-Bug: #1849870
Ieb4ab13cf8ca5683fcd7b18ed669e8a26659bff1 removed the upper-constraints
from the install_command which caused that only the test-requirements
are installed with the upper-constraints enforced. This caused that when
tox installed nova in the virtual env it installed the content of the
requirement.txt without enforcing the upper-constraints. Today networkx
2.4 package has been released to pypi. The taskflow lib depends on
networkx but does not pin the requirement but the openstack
upper-constraints pins the networkx requirements properly. Nova depends
on taskflow therefore when nova is installed by tox without the
upper-constraints the new networkx 2.4 is installed. This broke the nova
unit tests.
This patch makes sure that all the requirements are installed with the
upper-constraints enforced.
Change-Id: Iba797243d2a137b551223165a1af1a8676bcea02
Closes-Bug: #1848499
This is not encouraged by infra (particularly mordred and smcginnis)
since it's less obvious and easy to miss.
Change-Id: Ieb4ab13cf8ca5683fcd7b18ed669e8a26659bff1
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
- Add a new pdf-docs environment to enable PDF build.
- sphinxcontrib-svg2pdfconverter is used to handle SVG properly.
- maxlistdepth=10 in latex_elements is needed to handle
deeper levels of nesting.
- Sample config/policy files are skipped in the PDF document
as inline sample files cause LaTeX error [1] and direct links
in PDF doc is discouraged. Note that sample-config and sample-policy
need to be excluded to avoid the LaTeX error [1] and
:orphan: is specified in those files.
[1] https://github.com/sphinx-doc/sphinx/issues/3099
Change-Id: I3aaea1d15a357f550f529beaa84fb1a1a7748358
The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that Neutron publishes to api-ref/network, not networking anymore.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: Id2cf3aa252df6db46575b5988e4937ecfc6792bb
Sphinx 1.8 introduced [1] the '--keep-going' argument which, as its name
suggests, keeps the build running when it encounters non-fatal errors.
This is exceptionally useful in avoiding a continuous edit-build loop
when undertaking large doc reworks where multiple errors may be
introduced.
[1] https://github.com/sphinx-doc/sphinx/commit/e3483e9b045
Change-Id: Idc38751e1629947f5842651d99b1528f74247b42
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This one's actually important since it will be an error in future
versions of Python.
Change-Id: Ib9f735216773224f91ac7f49fbe2eee119670872
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
the current comment implies that setting
ignore_basepython_conflict simply silences
the warning emitted if the base python version
set in the gloabl [testenv] section conflicts
with the automatic python version selected form the
test env name .e.g. py27
this is not correct as the upstream documentation
states setting ignore_basepython_conflict=true
allow enforcing this rule that py37 implies python3.7
however when its is not set basepython from [testenv]
override the implied version form the env name.
https://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conflict
This change updates the comment to reflect that.
Change-Id: Ieab65a2a9c6309021d5ee91e56ac07889a9e9bc4
Related-Bug: #1832652
This finalizes the removal of the placement code from nova.
This change primarily removes code and makes fixes to cmd,
test and migration tooling to adapt to the removal.
Placement tests and documention were already removed in
early patches.
A database migration that calls
consumer_obj.create_incomplete_consumers in nova-manage has been
removed.
A functional test which confirms the default incomplete
consumer user and project id has been changes so its its use of
conf.placement.incomplete_* (now removed) is replaced with a
constant. The placement server, running in the functional
test, provides its own config.
placement-related configuration is updated to only register those
opts which are relevant on the nova side. This mostly means
ksa-related opts. placement-database configuration is removed
from nova/conf/database.
tox.ini is updated to remove the group_regex required by the
placement gabbi tests. This should probably have gone when the
placement functional tests went, but was overlooked.
A release note is added which describes that this is cleanup,
the main action already happened, but points people to the
nova to placement upgrade instructions in case they haven't
done it yet.
Change-Id: I4181f39dea7eb10b84e6f5057938767b3e422aff
This brings in a couple of new checks which must be addressed, many of
which involve a rather large amount of changes, so these are ignored for
now. A series of follow-up changes will resolved these.
'pycodestyle' is added as a dependency rather than it being pulled in
transitively. This is necessary since we're using it in tests.
Change-Id: I35c654bd39f343417e0a1124263ff31dcd0b05c9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
In the review of a similar change in placement [1], it was realized that
the nova lower-constraints tox job probably had the same problems.
Testing revealed this to be the case. This change fixes the job and
updates the related requirements problems accordingly.
The are two main factors at play here:
* The default install_command in tox.ini uses the upper_contraints.txt
file. When there is more than one constraints.txt they are merged and
the higher constraints win. Using upper and lower at the same time
violates the point of lower (which is to indicate the bare minimum
we are capable of using).
* When usedevelop is true in tox, the command that is run to install the
current projects code is something like 'python setup.py develop',
which installs a project's requirements _after_ the install_command has
run, clobbering the constrained installs. When using pbr,
'python setup.py install' (used when usedevelop is False) does not do
this.
Fixing those then makes it possible to use the test to fix the
lower-constraints.txt and *requirements.txt files, changes include:
* Defining 'usedevelop = False' in the 'lower-constraints' target and
removing the otherwise superfluous 'skipsdist' global setting to
ensure requirements aren't clobbered.
* Removing packages which show up in lower-constraints.txt but not in
the created virtualenv. Note that the job only runs unit tests, so
this may be incomplete. In the placement version of this both unit and
functional are run. We may want to consider that here.
* Updating cryptography. This version is needed with more recent
pyopenssl.
* Updated keystonemiddleware. This is needed for some tests which
confirm passing configuration to the middleware.
* Update psycopg2 to a version that can talk with postgresql 10.
* Add PyJWT, used by zVMCloudConnector
* Update zVMCloudConnector to a version that works with Python 3.5 and
beyond.
* Update olso.messaging to versions that work with the tests, under
Python 3.
* Adding missing transitive packages.
* Adjusting alpha-ordering to map to how pip freeze does it.
* setuptools is removed from requirements.txt because the created
virtualenv doesn't contain it
NOTE: The lower-constraints.txt file makes no commitment to expressing
minimum requirements for anything other than the current basepython.
So the fact that a different set of lower-constraints would be present
if we were using python2 is not relevant. See discussion at [1].
However, since requirements.txt _is_ used for python2, the
requirements-check gate job requires that enum34 be present in
lower-constraints.txt because it is in requirements.txt.
NOTE: A test is removed because it cannot work in the
lower-constraints context: 'test_policy_generator_from_command_line'
forks a call to 'oslopolicy-policy-generator --namespace nova' which
fails because stevedore fails to pick up nova-based entry points when
in a different process. This is because of the change to usedevelop.
After discussion with the original author of the test removal was
considered an acceptable choice.
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2019-03-05.log.html#t2019-03-05T13:28:23
Closes-Bug: #1822575
Change-Id: Ic6466b0440a4fe012731a63715cf5d793b6ae4dd
The api-samples tox target fails since it needs the
PlacementFixture which comes from the placement repo.
This fixes it by using the same deps trick as the other
functional test environments.
As a result, the shared environment is also removed.
Change-Id: I815b8b3cff5dcbb0fc4be5f1d85019111c2009e5
In tox versions after 3.0.0rc1 [1], setting the environment variable
PYTHONDONTWRITEBYTECODE will cause tox not to write .pyc files, which
means you don't have to delete them, which makes things faster.
In older tox versions, the env var is ignored.
If we bump the minimum tox version to something later than 3.0.0rc1, we
can remove the commands that find and remove .pyc files.
[1] https://github.com/tox-dev/tox/commit/336f4f6bd8b53223f940fc5cfc43b1bbd78d4699
Change-Id: I779a17afade78997ab084909a9e6a46b0f91f055
.pyc files are ignored by nova's .gitignore, as a result
if you change branches after the .pyc files are generated
for example from stable/ocata to master it does not remove
the .pyc files.
This can lead to the _validate_bytecode_header function that
is invoked as part of sphinx-build to fail as the .pyc
files no longer match the contents of the checked out files.
This change adds {[testenv]commands} to the docs envs
to clean up stale .pyc files before running sphinx-build
Change-Id: I536a260da639d32afbc998be220305f9489db375
The functional-* jobs have specific dependencies (external
placement) which need to be present in the job. When the
functional-py37 was created it conflicted with that change
but since it is not yet run in the gate, we didn't notice.
Also, because the functional jobs use the tox-siblings
functionality in the gate, the tox name and tox env have to
be the same (that's just how tox-siblings works), so that
is fixed here too.
Change-Id: I51e60a3149d653753ec4fc9d62a8c670e6eaf96e
Lee Yarwood