Yoga testing runtime is updated now
- https://governance.openstack.org/tc/reference/runtimes/yoga.html
which needs to test py38 and py39. Unit tests update are
handled by the job template change in openstack-zuul-job and
this commit makes changes to fucntional job to run py39 as voting
and updating the metdata in setup file.
Change-Id: I314fd61f20f2c3551f6231d191b7dcaaefabd2b5
Well, sort of. We enable them but immediately filter out the ones we're
actually seeing, the rationale being that we can address these in a
piecemeal fashion without the risk of introducing new issues.
There's a lot more to be done here. However, the work done in oslo.db
[1] should provide a guide for how to resolve the outstanding issues.
[1] https://review.opendev.org/q/topic:%2522sqlalchemy-20%2522+project:openstack/oslo.db
Change-Id: Iafe726d3819031c357460fd131bb2bb58babb4e2
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
autopep8 is a code formating tool that makes python code pep8
compliant without changing everything. Unlike black it will
not radically change all code and the primary change to the
existing codebase is adding a new line after class level doc strings.
This change adds a new tox autopep8 env to manually run it on your
code before you submit a patch, it also adds autopep8 to pre-commit
so if you use pre-commit it will do it for you automatically.
This change runs autopep8 in diff mode with --exit-code in the pep8
tox env so it will fail if autopep8 would modify your code if run
in in-place mode. This allows use to gate on autopep8 not modifying
patches that are submited. This will ensure authorship of patches is
maintianed.
The intent of this change is to save the large amount of time we spend
on ensuring style guidlines are followed automatically to make it
simpler for both new and old contibutors to work on nova and save
time and effort for all involved.
Change-Id: Idd618d634cc70ae8d58fab32f322e75bfabefb9d
As the bug and fix If71620e808744736cb4fe3abda76d81a6335311b showed
it is dangerous to forget instantiating the Mock class before it is
used in the test as changes on the class directly leaks out from the
test. In almost all the cases using Mock class directly is a bug and the
author original intention is to use an instance instead, just forgot
about the parents. So this patch adds two new hacking rules:
N367: catches the case when Mock class is aliased in the test:
self.mock_mystuff = mock.Mock
N368: catches when mock.patch instructed to use the Mock class as
replacement value during patching:
mock.patch('Bar.foo', new=mock.Mock)
For N367 the previous patch removed the last hit. For N368 this patch
removes the two hits exists.
Change-Id: Id42ca571b1569886ef47aa350369e9d2068e77bc
Related-Bug: #1936849
Some things that were missed in previous patches and are thrown together
here:
- Add alembic as an explicit dependency (we were getting it transitively
from oslo.db). We also bump the sqlalchemy dependency to a 1.4.x
release, which is the minimum supported by our chosen version of
alembic (more on this below)
- Remove tooling related to the old migrations
- Fix the tox whitelisting of the flaky MySQL tests
On the SQLAlchemy front, we opt for 1.4.13. Technically alembic should
support anything from 1.4.0, however, with SQLAlchemy >= 1.4.0, < 1.4.13
we see errors like the following in some tests:
sqlalchemy.exc.InvalidRequestError: Entity namespace for
"count(instance_mappings.id)" has no property "queued_for_delete"
There's nothing specific about this in the release notes for 1.4.13 [1]
but it definitely fixes things.
[1] https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.13
Change-Id: I4c8eb13f11aa7471c26a5ba326319aef245c9836
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This reverts commit 89e321b7b5.
Reason for revert: The previous change require develper to install multiple spcific version python
locally to run the test envs and it arbitrally pins lower constratins to the
oldest python we support but lower contratis shoudl work with the newwst python we support too.
on fedora or other operating systems that ship with default python version that exceed our supported
versions test can be run with tox -e py38 if required.
similarly our updated constratins shoudl work with the older python.
Change-Id: I358ff8128746c9d0c584a03381e55817c4aace3f
Declare specific versions of Python for environments that don't
specifically encode a version. We use the highest version officially
supported, Python 3.8, for most environments except lower-constraints,
which uses the lowest version, Python 3.6. This is necessary for users
on Fedora, where the default Python version is Python 3.9.
This will unfortunately introduce a small amount of busy work whenever
we change supported Python versions, as tox will need to be updated, but
that's a small price to pay for the usability wins that Fedora users
will see.
Change-Id: I240d5b8aa3eb1925b97af57e2644be61bfc106c2
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Depends-On: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/801773
This currently runs in the 'check' pipeline, as part of the pep8 job,
which causes otherwise perfectly valid backports to report as failing
CI. There's no reason a stable core shouldn't be encouraged to review
these patches: we simply want to prevent them *merging* before their
parent(s). Resolve this conflict by moving the check to separate voting
job in the 'gate' pipeline as well as a non-voting job in the 'check'
pipeline to catch more obvious issues.
Change-Id: Id3e4452883f6a3cf44ff58b39ded82e882e28c23
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Just removing mypy and then we will revert this patch once
I035d6dc752eaa83105cc12797765b304d843e1f7 is merged.
Change-Id: Iaa02e9d35419bf7100cef1014217210c1d5b6855
I4f551dc4b57905cab8aa005c5680223ad1b57639 introduced the environment
variable to disable the check-cherry-pick.sh script but forgot to allow
it to be passed into the pep8 tox env.
Change-Id: Ie8a672fd21184c810bfe9c0e3a49582189bf2111
This significantly speeds up our doc build process. This requires a
newer version of 'sphinx-feature-classification' and some tweaks to our
own in-tree extensions. While we're here, we drop the '-d DOCTREE_DIR'
parameter since it's of no use when we blast away our previously built
docs each time we build.
Change-Id: I679da65d44c40880f720df8a2f06286a19eb8d22
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Introduce API microversion 2.88, which makes the following changes to
a number of 'os-hypervisors'. Specifically, the following fields are
dropped from both the '/os-hypervisors/detail' (detailed list) and
'/os-hypervisors/{hypervisor_id}' (show) APIs:
- current_workload
- cpu_info
- vcpus
- vcpus_used
- free_disk_gb
- local_gb
- local_gb_used
- disk_available_least
- free_ram_mb
- memory_mb
- memory_mb_used
- running_vms
In addition, the '/os-hypervisors/statistics' API, which provided a
summary of the above stats but for all hypervisors in the deployment, is
dropped entirely.
Finally, the '/os-hypervisors/{hypervisor}/uptime' API, which provided a
similar response to the '/os-hypervisors/{hypervisor}' API but with an
additional 'uptime' field, has been removed in favour of including this
field in the primary '/os-hypervisors/{hypervisor}' API.
A small tweak to 'tox.ini' that allows us to share some venvs is
included.
Part of blueprint modernize-os-hypervisors-api
Change-Id: I515e484ade6c6455f82a3067940a418a0d7d965a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
In a heavily IO deprived CI VM the db migration tests could take a
significant amount of time and eventually time out. This patch moves the
tests into the same test executor worker process to spread the load
generated by these test in time until a final solution is found. For
example we hope that [1] will help eventually to decrease the load.
[1] https://review.opendev.org/q/topic:bp/compact-db-migrations-wallaby
Change-Id: I6ce930fa86c82da1008089791942b1fff7d04c18
Related-Bug: #1823251
Nova does not run gabbi test since placement is moved to a separate
git repository. So the gabbi related tox.ini comment is removed.
Change-Id: Ic324e3e32fa03478895b32fa583e805ee6c721e2
This patch adds both a tox target and a zuul job to run functional tests
with python3.9
Depends-On: https://review.opendev.org/760932
Change-Id: I672904e9bfb45a66a82331063c7d49c4bc0439df
This linting makes building docs an even more painful process than it
would otherwise be. We already do this as part of the pep8 target,
which the gate runs, so there's no need to do this here.
Change-Id: I36ae872dd21299ad5d165422cb83564eafd89bea
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Add the following hacking rule.
* N366: The assert_has_calls is a method rather than a variable.
Not correct: mock_method.assert_has_calls = [mock.call(0)]
Correct: mock_method.assert_has_calls([mock.call(0)])
This patch is a follow-up patch for
Id094dd90efde09b9a835d4492f4a92b8f8ad296e.
Change-Id: I892f8c23ee44f2b3518776a9705e3543f3115cae
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
These will not be used in a world without the XenAPI driver.
Change-Id: I5bc3c7855b817c4ce2b8919c76be80cceabeec9e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The Python 2.7 Support has been dropped since Ussuri.
So remove hacking rules for compatibility between python 2 and 3.
- [N325] str() and unicode() cannot be used on an exception.
Remove or use six.text_type()
- [N327] Do not use xrange(). xrange() is not compatible with Python 3.
Use range() or six.moves.range() instead.
- [N344] Python 3: do not use dict.iteritems.
- [N345] Python 3: do not use dict.iterkeys.
- [N346] Python 3: do not use dict.itervalues.
See also line 414 in https://etherpad.opendev.org/p/nova-victoria-ptg
Change-Id: If4335b2e8ef5bbabba37598110c1aa8269635c2f
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
This adds a tools/ script that checks any cherry-picked hashes
on the current commit (or a provided commit) to make sure that
all the hashes exist on at least master or stable/.* branches.
This should help avoid accidentally merging stable backports
where one of the hashes along the line has changed due to conflicts.
Change-Id: I4afaa0808b75cc31a8dd14663912c162281a1a42
The N319 check previously asserted that debug-level logs were not
translated. Now that we've removed all log translations, we can
generalize this to all logs. We reuse the same number since these
numbers are really just metadata and not public contracts.
This also allows us to update the N323 and N326 checks, which ensure we
import the translation function, '_', wherever it's used and don't
concatenate translated and non-translated strings. Since we're no longer
translating logs and the '_LE', '_LW' and '_LI' symbols are no longer
provided, we don't need to consider logs in either of these cases.
Change-Id: I64d139ad660bc382e8b9d7c8cd03352b26aadafd
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
mypy is an experimental optional static type checker for Python that
aims to combine the benefits of dynamic (or "duck") typing and static
typing. While still in development, most features are supported and it's
already being used by real world projects like Sphinx. Let's start small
by integrating it into some of the interfaces that nova exposes. We can
further build upon this if it works out.
This change sets up the boilerplate necessary to use mypy in nova. Type
annotations are not included for any module - these will be added
separately. We're calling mypy by way of a script, as this allows us to
store a list of files that we have converted while we're in the process
of adding type annotations where necessary.
Change-Id: I75ab46a6768c4ca2050fdde2b7f8eeb90724c8c6
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
As per Victoria testing runtime[1], we need to tests py3.6,
py3.7, and py3.8.
- py3.7 is being tested with integration jobs.
- py3.6 and py3.8 are tested with unit test jobs.
Nova functional tests are testing py3.6 which can be moved to the latest
python available in this cycle which is 3.8. We do not need to run functional
or integration tests on each supported python version. Testing them on
the latest python version is enough. Coverage of all supported python version
is achieved via unit tests job.
[1] https://governance.openstack.org/tc/reference/runtimes/victoria.html
Change-Id: I1d6a2986fcb0435cfabdd104d202b65329909d2b
nova-tox-functional is py2 job and need placement as required project.
and started failing for incompatible py version:
ERROR: Package 'openstack-placement' requires a different Python: 2.7.17 not in '>=3.6'
- https://zuul.opendev.org/t/openstack/build/b460a8c59ad64e57b871aa3c00638b01/log/job-output.txt#782
Placement master is now py3 only which mean master version will stop
working on py2 env:
- I6f458fb60b5a33b5aa2ce3ab292862ab98eb4670
Current tox env mention the placement master link as deps, where need to use
the stable version for stable jobs. Making deps with version not the
fresh git clone like how other test-requirements.txt deps work.
Change-Id: I96bb9fcb55f2a4fc3aefa01db70b45740db166c8
Add the validation framework necessary to verify extra specs along with
the definitions for every extra spec we currently recognize in-tree.
None of this is currently used since we don't have the API microversions
wired up, but that will come in a future patch.
Note that we must add the H238 hacking check to the ignore list here,
since this includes our first use of Python 3-type classes without the
explicit 'object' subclass. This can be removed when that check is
removed from hacking.
Part of blueprint flavor-extra-spec-validators
Change-Id: Ib64a1348cce1dca995746214616c4f33d9d664bd
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This bumps the version of flake8 and pycodestyle to something much
newer, which resolves a long-standing warning about nested sets and
allows us to use new fangled features like f-strings if we so choose.
Change-Id: I0bb9077f1cea2243b7945e87cfa140f9cf89d558
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
It's Ussuri. We can *finally* stop testing Python 2 [1]. Time to party.
We don't attempt any cleanup but simply stop testing with Python 2,
indicate that we only support Python 3 via 'setup.cfg' and remove any
Python 2 only dependencies. Our 'tox.ini' is modified such that
'functional' now runs with 'python3', whatever that may point to, though
the gate will only use a versioned variant (currently
'functional-py36').
This should free up a significant amount of resources from the gate and
let us start using Python 3 idioms in our code. Win-win.
[1] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html#python2-deprecation-timeline
Change-Id: Ie1a0cbd82a617dbcc15729647218ac3e9cd0e5a9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Change Iba797243d2a137b551223165a1af1a8676bcea02 was a bit
overzealous in using {[testenv]deps} and changed the docs
tox target to also install requirements.txt and
test-requirements.txt which means for docs builds we're
installing things like psycopg2 which we shouldn't be doing
if we don't have the correct native packages installed to
build those types of dependencies.
Change-Id: Ib718911596b93ec6ec7e899210300d2f0d9572ed
Closes-Bug: #1849870
Ieb4ab13cf8ca5683fcd7b18ed669e8a26659bff1 removed the upper-constraints
from the install_command which caused that only the test-requirements
are installed with the upper-constraints enforced. This caused that when
tox installed nova in the virtual env it installed the content of the
requirement.txt without enforcing the upper-constraints. Today networkx
2.4 package has been released to pypi. The taskflow lib depends on
networkx but does not pin the requirement but the openstack
upper-constraints pins the networkx requirements properly. Nova depends
on taskflow therefore when nova is installed by tox without the
upper-constraints the new networkx 2.4 is installed. This broke the nova
unit tests.
This patch makes sure that all the requirements are installed with the
upper-constraints enforced.
Change-Id: Iba797243d2a137b551223165a1af1a8676bcea02
Closes-Bug: #1848499
This is not encouraged by infra (particularly mordred and smcginnis)
since it's less obvious and easy to miss.
Change-Id: Ieb4ab13cf8ca5683fcd7b18ed669e8a26659bff1
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
- Add a new pdf-docs environment to enable PDF build.
- sphinxcontrib-svg2pdfconverter is used to handle SVG properly.
- maxlistdepth=10 in latex_elements is needed to handle
deeper levels of nesting.
- Sample config/policy files are skipped in the PDF document
as inline sample files cause LaTeX error [1] and direct links
in PDF doc is discouraged. Note that sample-config and sample-policy
need to be excluded to avoid the LaTeX error [1] and
:orphan: is specified in those files.
[1] https://github.com/sphinx-doc/sphinx/issues/3099
Change-Id: I3aaea1d15a357f550f529beaa84fb1a1a7748358
The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that Neutron publishes to api-ref/network, not networking anymore.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: Id2cf3aa252df6db46575b5988e4937ecfc6792bb
Sphinx 1.8 introduced [1] the '--keep-going' argument which, as its name
suggests, keeps the build running when it encounters non-fatal errors.
This is exceptionally useful in avoiding a continuous edit-build loop
when undertaking large doc reworks where multiple errors may be
introduced.
[1] https://github.com/sphinx-doc/sphinx/commit/e3483e9b045
Change-Id: Idc38751e1629947f5842651d99b1528f74247b42
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This one's actually important since it will be an error in future
versions of Python.
Change-Id: Ib9f735216773224f91ac7f49fbe2eee119670872
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Ghanshyam Mann