There is a race between where Xen creates the domain and starts logging
and when XAPI returns to say that the domain has started.
If the cronjob to rotate the guest logs runs in this period, it can
delete the console logs before we set the last_dom_id which is used to
preserve logs.
This change adds a fixed delay before deleting any log files, allowing
the VM start operation to return after the domain has been successfully
created.
Change-Id: I82ae3e008974f33b60256366f171abf1f0369e60
Closes-Bug: 1535616
Misspelling in the message:
"You have uncommited changes"
Should be: "You have uncommitted changes"
Totally 2 occasions in Nova base code.
Change-Id: If0ad3163f3b9db3f31f7e9fa40245282c1a0acb4
It's possible that a log file is created while this script is
running, and if being created after we get the valid last_dom_ids
then the log file may be deleted incorrectly.
Also add some logging which can be used to trace any future race
conditions
Change-Id: I1df1a0775a7dae41105d6c41b046c86002eb5eec
Closes-Bug: 1516721
Currently 'test_list' variable is being checked
for empty list like 'if test_list == []:'.
The more standard approach is like 'if not test_list:'.
TrivialFix
Change-Id: I0b389c11b992c93529b305b1ebce6b181f9a9a30
Fixes cache cleaner script by removing import of
driver and using session directly to call vm_utils.
There is no test included for this. Since this is out of nova
tree, adding tests is tracked in 1481705
Closes-Bug: #1481693
Change-Id: I25ee22777baecaf64de63a39d19e8f8f514f70b3
The 'tox -epep8' command takes a long time as it checks
every single file in the Nova git repository (~1,400 at
time of writing).
This makes tox use a simple wrapper around flake8 which
can be told to restrict the check to only files changed
in the current command. This can be invoked in a simple
manner with 'tox -epep8 -- -HEAD'. Since most commits
only touch a handful of files, this will usually be
far faster than checking all 1,400 source files.
To check an entire branch for bisectability it can be
automated via
git rebase -i master -x 'tox -epep8 -- -HEAD'
Change-Id: I157d1ccb883ca02402eee51fd7d6a50f86079389
The sphinx latex generation generates invalid latex that won't compile
when you try to use it.[1][2] This commit adds a helper script to generate
the sphinx latex and then modify it so it'll work. It depends on
ImageMagick convert and sed being available to work.
[1] https://github.com/sphinx-doc/sphinx/issues/1907
[2] https://github.com/sphinx-doc/sphinx/issues/1959
Change-Id: Id289c10907aaddae2483f18b39063852ec699d66
Notes on migration changes/tests:
1. The initial havana migration does not create any foreign keys which
require the instances.uuid column since it's nullable and DB2 won't
allow creating a unique constraint over a nullable column. To create
the foreign keys on instances.uuid, this change depends on commit
e07a2b2d4be0503ad8dad75daa632be27cf83320 to make instances.uuid
non-nullable and create a unique constraint on that column. Then
the new migration here creates the missing instances.uuid
related foreign keys for DB2.
2. Commit b930fb3a6b changed
test_migrations.py to use oslo.db's opportunistic test fixture
which currently only supports sqlite/mysql/postgresql. Unit test
support for DB2 therefore needs to be added to oslo.db and then
hooked into nova, but that's targeted for the 2016.1 'M' release
so is not part of this change.
3. The 247_nullable_mismatch migration is updated for DB2 because
making the pci_devices.deleted column nullable=True fails since
the column is in a UniqueConstraint already (DB2 would require
it to be in a unique index instead for the column to be nullable).
There is a foreign key involved (created in 246) so for DB2 to
have the same FKey it requires a UC and that requires the deleted
column to be non-null, which doesn't work with the SoftDeleteMixin.
So in this case the schema will diverge for DB2 until we make
the deleted column non-nullable for all tables.
4. The 252_add_instance_extra_table migration is updated for DB2
because a foreign key is created between the instance_extra
table and the instances.uuid column (which is nullable at the
time) so we have to handle that foreign key constraint creation
in the 296 migration with the other foreign keys to
instances.uuid.
5. The 271 migration is updated to add ibm_db_sa the same as
sqlite and postgresql since it works the same in this case for
DB2.
6. The API migrations need checks for DB2 to avoid a duplicate index
SQL0605W error since DB2 implicitly creates an Index when a
UniqueConstraint is created and will fail if trying to add a
duplicate index on the same column(s) that are in the UC.
Implements blueprint db2-database
Change-Id: Ic4224e2545bcdfeb236b071642f9f16d9ee3b99f
As discussed in the Liberty Design Summit "Moving apps to Python 3"
cross-project workshop, the way forward in the near future is to
switch to the pure-python PyMySQL library as a default.
https://etherpad.openstack.org/p/liberty-cross-project-python3
Change-Id: I590c26523f4dc8d14c45e61bb89555486629b64d
Moving console logs to a temporary file is a clear race when Nova tries to
access a file it might not be there.
We need to keep the readable file in place at all times. Atomic operations
like mv are fine as if the file was opened before the mv operation Linux
will keep the contents available until the file is closed by Nova's plugin.
Futher, we need to be able to report the console logs from domains that have
been shutdown but still exist on the host. Ensure we keep track of the domain
that we last booted with
Change-Id: Icfff9886d79f788be871bf28c8b729a57dcd94d2
Fixes-bug: 1466914
Oslo team is actively getting rid of the old style generator.py
in openstack/common/config. So we should switch to the newer
oslo-config-generator which is the way forward. For simplicity's
sake, i used one single list_opts in opts.py. However we can
split this up and add entrypoints in setup.cfg if we decide to
do that instead.
blueprint oslo-config-generator
Change-Id: Ib7c72af7be47df08983708ac6da0ce103518b78c
Iterating on a dictionary iterators on its keys, so calling its
iterkeys() method is not needed.
Moreover, the dict.iterkeys() method is gone in Python 3. This change
makes the modified code compatible with Python 3.
Blueprint nova-python3
Change-Id: I140f499d2054bb7d5486e4c61a2f886be6649cbb
MODULEPATH is apparently a well known variable from
environment-modules package. So let's just use another
one as we don't care about the specific var name itself.
Closes-Bug: #1437904
Change-Id: Id79bbdaad498c771dc4c032256bd8160b5663ef9
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
This commit includes:
- using oslo_utils instead of oslo.utils
- using oslo_serialization instead of oslo.serialization
- using oslo_db instead of oslo.db
- using oslo_i18n instead of oslo.i18n
- using oslo_middleware instead of oslo.middleware
- using oslo_config instead of oslo.config
- using oslo_messaging instead of "from oslo import messaging"
- using oslo_vmware instead of oslo.vmware
Change-Id: I3e2eb147b321ce3e928817b62abcb7d023c5f13f
testr run --until-failure --subunit does not actually fail when a test
fails, upstream bug filed at
https://bugs.launchpad.net/testrepository/+bug/1411804
Work around this by reverting to old style version when running in
--until-failure mode.
Change-Id: Ic2b26c423e4e6c729bb1357bc901fdf1afdb083f
The pylint tests have been broken for several months and
the Jenkins job has also been disabled. There has been
no sign of attempts to address this, so just delete what
remains of the pylint support so developers don't mistakenly
think it is supposed to work with patches they are submitting.
Change-Id: Ifacc99e45acb0e9a72be6c31113e0cf073354f02
This commit removes the local copy of subunit-trace in nova and uses
the packaged version of the utility in tempest-lib.
Change-Id: I1ed2fea9ff3d8194628b722684492f8ba7a3cabb
Make sure we include database and concurrency options in the
sample config file.
Change-Id: I66e68ec120cba2cee86e36aa26720199876c40a4
Closes-Bug: #1391782
python-novaclient had some 6 month old stalled patches, add it to the
overall abandon cleanup policy.
Change-Id: I28831acc1c0cee846e79cc63138b7bc2ce8ee03d
This patch fixes the pep8 issues of the tool scripts and
configures tox to run pep8 on the tools directory.
Two hacking rules have been modified to ignore the tool directory as both
rules are checking oslo usage which is not critical in case of tools
- N310 use timeutils instead of datetime module
- N324 use jsonutils instead of json module
The tools/xenserver directory is still excluded from pep8 as it contains
scripts that are expected to run on dom0 with python2.4 therefore the
following rules cannot be meaningfully followed
- H231 Python 3.x incompatible 'except x,y:' construct
- H233 Python 3.x incompatible use of print operator
Change-Id: Icfd9e708a2ca1e421676b410e41807b2e630718f
This is an automated script to abandon the reviews that we agreed as a
community are beyond stale.
No activity for 4w with a -2 on them or No activity for 4w and Jenkins
doesn't have a +1 or better on it.
Can be run by any core reviewer.
Change-Id: I19636f04b691c0b74af306709d54b26978c19282
Pareseable output format and --include-ids parameter are deprecated
and ignored in the pylint version used on the gate.
These functions are replaced by the --msg-template parameter.
This patch changes the runner to use the new parameter so that the
job can run properly on the gate
Closes-Bug: #1394606
Change-Id: I1de424bae4c2b634cd0bc804cb7e81624c96a195
Instead of passing lockutils loading explicitly in the test run
command line, provide the support for individual test classes to ask
that lock_path is setup with the REQUIRES_LOCKING = True class
parameter.
Augment all classes that currently fail without this with
REQUIRES_LOCKING = True. Hopefully many of those can be individually
removed in the future so we can get out of needing external locking in
any of our unit tests.
Change-Id: I56229e93ad61ac823646a4119a035d36c3400173
This makes subunit-trace error out if it finds that there were no test
results in the subunit stream. This guards against the case where
something breaks the test toolchain and we stop running tests
entirely. Copied over from equivalent code in Tempest where this has
served us well.
Change-Id: I765395335a68dfc7eb754317f8c45c3dfdd85949
This patch made unit tests stop running (always runs 0 unit tests) we
should revert and put a check in place to prevent that from happening
so that we don't regress the rest of Nova trying to figure out the
oslo.concurrency issue.
This reverts commit 382294762b.
Change-Id: I98b04ac2a0ba67ba8af44867799317fdf95b8aff
Changes include:
55ca7c3 Split cliutils
1b44601 Allow dictionary lookup in credentials with dot notation
6c706c5 Delete graduated serialization files
3edbfb3 remove caching param from prettytable call
9ce1d96 Fix i18n import
5d40e14 Remove code that moved to oslo.i18n
a4be4ed Remove graduated config modules
6ff6b4b Switch oslo-incubator to use oslo.utils and remove old modules
70189c9 Remove graduated test and fixtures libraries
8b71f57 Delete graduated db files
6ede600 rpc, notifier: remove deprecated modules
6b65a41 Fix build break - switch from str to hash for lookup
f76f44c Delete the token and endpoint on expiry of token of client
aebb58f Fix typo to show correct log message
1131b56 Enabled mask_password to handle byte code strings
6b048e7 Let oslotest manage the six.move setting for mox
Change-Id: Ic588aec0fc0291440feaac3439836054bd3608cd
oslo.utils library now provides the functionality previously in
oslo-incubator's excutils, importutils, network_utils, strutils
timeutils, units etc. Some modules already moved to oslo.utils
will still be around since other code in nova/openstack/common/
are using it and will be removed in a subsequent commit.
Change-Id: Idc716342535fdfa680963e0e073ddb46f5f1eb34
In order to get similar behavior to the way testr out of the box
works, only displaying failing results, all the user to define
TRACE_FAILONLY environment variable (any value will do). After this
point only failures, and things leaking out of stderr will end up on
the screen.
We also need to remove the suppression of the inline fail dumps for
this to have the impact that you'd expect.
Change-Id: I72911adc8a03450522429c52db661ec0a1bc4678
Tempest developed a set of wrappers around testr to print out tests as
they are running through using the subunit stream. This also displays
which workers the various tests are running on. Adapted for nova's
need for loading the locking.
I expect that the needs of nova will diverge this over time,
especially what's displayed during tracing, as such I think it's fine
to just do a transplant and not try to turn this into some external
repo.
Change-Id: Ied1708515fc49f4e18cd8dbb7b2cd821f85e1966
The auth_token middleware was moved from keystoneclient.middleware
to keystonemiddleware. The config generator is configured to point
to the new location.
Closes-Bug: #1366594
Change-Id: I5543973b2c7ac06c048686c24354ecf6eb251027
oslo.i18n provides the i18n functions that were provided by
oslo-incubator's gettextutils module. Some tests that were
using internal details of the library were removed.
Change-Id: I44cfd5552e0dd86af21073419d31622f5fdb28e0
CLI options must be registered before the command line is parsed or a
cfg.ArgsAlreadyParsedError is raised.
Since commit 2cbea24 we are registering two 'remote_debug' arguments
in nova.service. In commit b506529, we see a hack being added to
ensure that nova.service is imported and the CLI opts registered
before config.parse_args() is called.
To clean this up somewhat, refactor all the elements of this remote
debugger support into nova.debugger. Avoid importing modules at
module import time since we need to import this before monkey
patching. Add a function for registering the CLI options just before
config.parse_args() is called. And fail gracefully if the CLI options
aren't registered when we called debugger.init().
Note that because the options aren't registered at module import time
any more, the config generator doesn't pick them up. However, that's
actually a good thing since this support doesn't work if you enable it
via config files since debugger.enabled() only looks at the command
line.
DocImpact: remote_debug options are not available via the config file,
only via the command line.
Change-Id: I97f747a2fb9222137203657df1d86ba89f3219e2
This file can be installed to properly configure the
firewall to allow VNC access to openstack instances
DocImpact
Closes Bug: 1294411
Change-Id: Ifc9a7d88c71dfe1b66d6ac395d9574cc619c1d3a
Added missing imports along with adding a call to config.parser_args as needed
to allow vm_vdi_cleaner to work against current master.
Corrected spelling error in comment.
Change-Id: I4d30d0f7f9718cb29fb8c4c18e9e6fa23f3496ef
Closes-Bug: #1279942
We don't need to have the vi modelines in each source file,
it can be set in a user's vimrc if required.
Also a check is added to hacking to detect if they are re-added.
Change-Id: I347307a5145b2760c69085b6ca850d6a9137ffc6
Closes-Bug: #1229324
The oslo.messaging library takes the existing RPC code from oslo and
wraps it in a sane API with well defined semantics around which we can
make a commitment to retain compatibility in future.
The patch is large, but the changes can be summarized as:
* oslo.messaging>=1.3.0a4 is required; a proper 1.3.0 release will be
pushed before the icehouse release candidates.
* The new rpc module has init() and cleanup() methods which manage the
global oslo.messaging transport state. The TRANSPORT and NOTIFIER
globals are conceptually similar to the current RPCIMPL global,
except we're free to create and use alternate Transport objects
in e.g. the cells code.
* The rpc.get_{client,server,notifier}() methods are just helpers
which wrap the global messaging state, specifiy serializers and
specify the use of the eventlet executor.
* In oslo.messaging, a request context is expected to be a dict so
we add a RequestContextSerializer which can serialize to and from
dicts using RequestContext.{to,from}_dict()
* The allowed_rpc_exception_modules configuration option is replaced
by an allowed_remote_exmods get_transport() parameter. This is not
something that users ever need to configure, but it is something
each project using oslo.messaging needs to be able to customize.
* The nova.rpcclient module is removed; it was only a helper class
to allow us split a lot of the more tedious changes out of this
patch.
* Finalizing the port from RpcProxy to RPCClient is straightforward.
We put the default topic, version and namespace into a Target and
contstruct the client using that.
* Porting endpoint classes (like ComputeManager) just involves setting
a target attribute on the class.
* The @client_exceptions() decorator has been renamed to
@expected_exceptions since it's used on the server side to designate
exceptions we expect the decorated method to raise.
* We maintain a global NOTIFIER object and create specializations of
it with specific publisher IDs in order to avoid notification driver
loading overhead.
* rpc.py contains transport aliases for backwards compatibility
purposes. setup.cfg also contains notification driver aliases for
backwards compat.
* The messaging options are moved about in nova.conf.sample because
the options are advertised via a oslo.config.opts entry point and
picked up by the generator.
* We use messaging.ConfFixture in tests to override oslo.messaging
config options, rather than making assumptions about the options
registered by the library.
The porting of cells code is particularly tricky:
* messaging.TransportURL parse() and str() replaces the
[un]parse_transport_url() methods. Note the complication that an
oslo.messaging transport URL can actually have multiple hosts in
order to support message broker clustering. Also the complication
of transport aliases in rpc.get_transport_url().
* proxy_rpc_to_manager() is fairly nasty. Right now, we're proxying
the on-the-wire message format over this call, but you can't supply
such messages to oslo.messaging's cast()/call() methods. Rather than
change the inter-cell RPC API to suit oslo.messaging, we instead
just unpack the topic, server, method and args from the message on
the remote side.
cells_api.RPCClientCellsProxy is a mock RPCClient implementation
which allows us to wrap up a RPC in the message format currently
used for inter-cell RPCs.
* Similarly, proxy_rpc_to_manager uses the on-the-wire format for
exception serialization, but this format is an implementation detail
of oslo.messaging's transport drivers. So, we need to duplicate the
exception serialization code in cells.messaging. We may find a way
to reconcile this in future - for example a ExceptionSerializer
class might work, but with the current format it might be difficult
for the deserializer to generically detect a serialized exception.
* CellsRPCDriver.start_servers() and InterCellRPCAPI._get_client()
need close review, but they're pretty straightforward ports of code
to listen on some specialized topics and connect to a remote cell
using its transport URL.
blueprint: oslo-messaging
Change-Id: Ib613e6300f2c215be90f924afbd223a3da053a69
Commits included:
e3dddd7 generator: use EXTRA_* env vars in the bash script
e8e636c generator: add an EXTRA_LIBRARIES env variable
6da13e8 generator: rename EXTRA_MODULES_FILE to RC_FILE
5dce17b Use entry points to discover options in libraries
dd9aa2b Remove unused variables
ad17a69 Fix filter() usage due to python 3 compability
343686b Add check_uptodate to tools/config
The library option discovery support is the main addition we're
interested in since it is required for oslo.messaging.
Change-Id: I68958ddd1c1d0c02b54002c36b97270efaaaa892
Sync from oslo-incubator with the following patches about the config
generator:
7c18261 Temporary workaround for config generator
4b3e32b Sort the output of config/generator.py by group name
806272e Improve error reporting on import failures
6cd1c33 Clean up extra modules code
3d077a3 Print the deprecated group/name for each option
aec6d94 config.generator: allow extra modules importing
ddc4f0d Fix error when no custom config file matches *.conf.sample
d4bf84d Allow generate_sample MODULEPATH to be set in env
UpgradeImpact
DocImpact, the default values in keystone_authtoken config group are
changed.
Change-Id: If7181e14db2e43e42255bb6221d0543c13c4e5a6
Bob Ball