We can hit issues with corrupted VHDs if we copy a VHD while XenServer
is performing other operations. This happens because there are times
when we copy the VHD chains while XenServer is still performing a
coalesce of the VHD chain.
In most cases, vhd-util should be able to safely repair any metadata
corruption. It can copy the copy of the VHD footer at the front of the
VHD file and add it at the bottom on the VHD file. There is no VM data
loss, due to the way the coalesce happens, but the chain will be bigger
than it would be both before and after the coalesce.
This does not, however, ensure that snapshots are valid before uploading
them to glance. But should you launch a corrupted snapshot, this change
would fix up the snapshot, and allow it to boot correctly.
Closes-Bug: #1362595
Change-Id: I88b737d7e97964a9db5ccf2c39dea7fd0701ead4
During chunk upload if a subprocess exception is thrown, a malformed
chunk upload occurs.
This patch adds logging for subprocess exceptions and sends the
terminator even in the event of a subprocess exception. The incomplete
file upload will then be cleaned up by the client.
Change-Id: I559f201430bcd4c36d669c7772a10bfe3cd54d41
Closes-Bug: 1365637
If Glance returns a 500 response on an initial attempt to upload
a snapshot image, it will set the image status to killed. Any retry
attempts will fail with a 409 response.
Hence, Nova should not attempt to retry the upload in this case, which
will eliminate many pointless retries.
Closes-Bug: 1349933
Change-Id: I1a8f2416923a368b02cf8963c747ebbb24d749a1
pep8 E265 makes sure block comment start with '# '. Fix and gate on this
new rule as it helps improve comment readability.
In the few cases where the comment was just commented out code, remove
the comment.
Change-Id: Iea1c445df8ddc2b6c17a4ab697ad756eef2f91fa
Both H305 and H307 are part of the OpenStack style guide.
Ensure that there are 3 groups of imports, stdlib,
third-party and project specific. Within each category
ensure that imports are in alphabetical order.
Change-Id: Id4994551c135c3c1a4982153f0c6cacba6176b95
This commit addresses some optimizations when using libtorrent to
download. The settings are documented here:
http://www.rasterbar.com/products/libtorrent/reference-Settings.html
I'm proposing to add these with dom0 environment overrides in case
any other adjustments are needed. The changes effectively double the
download rate from the default settings. It also allows unbalanced
download/upload which is preferred in openstack environments.
No unit tests were added as plugins are not currently unit tested.
Closes-bug: #1303993
Change-Id: I503a6a521c196e4ad58e0a16aba2988f66a9cdca
Using ssh depends on the host being set up to be able to SSH into
itself which is not a common scenario. While this is unavoidable for
the current implementation of resize across multiple hosts, if there
is a single host (i.e. a test scenario) or the resize is restricted
to the same host then we can succeed without SSH access
Dependency on Ia310e31d31aaf5c979e41c64af8223202a18e03a is because the
tests will always fail without Ia310 therefore this fix cannot be tested
without taking Ia310 as well.
Closes-bug: 1308064
Change-Id: I15802a1d97d380b1c5b74fc9f92ece2494fe789a
Currently, when the network connection to glance is interrupted, there
is different behaviour on upload and download.
This change ensures the behaviour between the two code paths is more
consistent.
Uploads generally need to be given more time before they timeout, so
to keep a single timeout between upload and download, the timeout is
increased to 90 seconds.
At the same time, it ensures the tar process gets killed when any
issues occur with the communication between the hypervisor and
glance.
Change-Id: Id5396e5d3c1052dc2979476a886412da65e08670
Closes-Bug: #1284596
Fixes bug 1286187
Neither execute_get_output() nor execute() would wait for the process to
fully finish executing. This could potentially create a race condition
where commands execute in a different order than intended.
Also, for execute() it could cause a process to never finish executing if
it generates enough output to block writing to the pipe.
Change-Id: I3404f4b3ca1cddeec2f3e7b393817a6ccc42bec7
This check indicates on comments in which multi line docstring should
start without a leading new line. This change fixed all violators of
said check.
Change-Id: Ic7357b8c7420767dba611f6fcee07b7700f3aea8
We added a function into the xenhost plugin to get information
about PCI devices. Roughly we run the lspci command on dom0. This
information will be used to get the list of pci devices that are
passed on the pciback.hide dom0 command line.
The hide option is used to hide the devices from the normal guest
drivers and assign them to the pciback kernel driver at boot on dom0
instead of their normal driver. We will parse the output of the lspci
command to find which device is using the pciback kernel driver and
thus to know if it has been passed to the pciback.hide option.
This information will be used to perform the match with the
list of pci devices provided in pci_whitelist into /etc/nova/nova.conf.
Implements: blueprint pci-passthrough-xenapi
Change-Id: I465fc5d29f3c47ab0079adcfcc2d7d6501bd4b20
Add a new host plugin to monitor whether the garbage collector is running.
This is then used by the wait_for_vhd_coalesce function to know whether
waiting any more is likely to produce the expected results.
Increased the number of times we loop waiting for coalesce as the loop
now has knowledge that something is still happening.
DocImpact
Change to parameter default.
Change-Id: Idcc0738945bd1aee8cdb52c9cfabd798f85c31db
Closes-bug: 1258169
We don't need to have the vi modelines in each source file,
it can be set in a user's vimrc if required.
Also a check is added to hacking to detect if they are re-added.
Change-Id: I347307a5145b2760c69085b6ca850d6a9137ffc6
Closes-Bug: #1229324
It seems there are cases when the server gets stuck in a building state
because the download from glance fails, but the nova-compute doesn't
find out.
After some code inspect, a likely cause seems to related to having no
socket timeout inside urllib2 when we download the image from glance. A
workaround for this issue is to configure the default socket timeout as
described here:
http://docs.python.org/3.1/howto/urllib2.html#sockets-and-layers
Fixes bug 1251650
Change-Id: I8b34d95d3524f3825b28e61cbbb59290653d3711
Fix the agent update message send so Windows guests will work
Message must be formatted: "url,md5sum"
Change-Id: I675cc5a7ca522696eab1820c14c48361ff7e0f74
Closes-Bug: 1250165
Nova shouldn't be building any types of packages. And if we are we should
be testing this code in gate somewhere.
Change-Id: I70d82a6173b5eda183d4e49d330ae97dd14f9078
Because the plugins live on a host seperate to Nova we need an interface
to test whether they are the expected version. We can't use pip or other
requirement systems as they are cross-machine.
Closes bug 1226622
Change-Id: I58ab669061f51bd87071e2cf0d93d33021001309
Due to some issues in the gate, several flake8 errors got merged
recently. This patch fixes the most recent issues found.
For the H304 relative import errors, we have to skip those because they
are put into /etc/xensource/scripts which is the current working
directory when running the plugin.
For the H231 incompatible python 3 'except x,y' construct, we have to
skip those because this code is written to run on python 2.4 and 'except
x as y' does not work with python 2.4.
Note that in cleaning up some of the H304 failures for relative imports,
I also re-arranged the imports to follow the hacking guide of doing
standard library packages first, then third party packages, and finally
nova-specific packages.
Closes-Bug: #1229753
Change-Id: I1c2211fd6a10d43d7e65cdb4e18530397788cf2c
* Removes dead code
* Prefixes internal functions with underscore
* Removes a few one-off functions in favor of inlining
Change-Id: Ic2f8c2e512df270f3b1b0e35ded98832a5152c72
Due to a previous commit, a client of make_subprocess is making a call
with an env parameter. As make_subprocess does not have such a
parameter, image upload fails. This patch adds the env keyword argument
to make_subprocess, and use that in the Popen call.
Fixes bug 1216267
Change-Id: I8098b484067b54c15fe95f2bf84c5890db7ebcb3
This commit adds the instance uuid to the rsync debug output in
the debug logs. The issue we were seeing is that when multiple
migrations are ongoing at the same time, it's not easy to
track the rsync progress of each migration.
Fixes: bug #1215629
Change-Id: Iec662bdc0aed1c6f54ecaa062335dc5504ac030f
Currently only used by XenAPI.
Sets GZIP environment variable before spawning tar -cz to force the
compression level. Defaults to none, which means the GZIP environment
variable is not set and the default (usually -6) is used.
Implements: blueprint image-compression-mode
Change-Id: I02e136b7215a4f2cefc259f40bde9c1b205737ed
This patch adds support for a new kind of Glance image, a specially crafted ISO
which supports iPXE booting, giving customers a means to roll their own image.
Two virt-layer modifications were needed. The first was adding configurations
for the iPXE ISO feature (network to use, boot menu, mkisofs_cmd). The second
was the ability to inject networking info into the ISO after it was
downloaded.
To use this feature, operators should enable the `ipxe_boot` image-property.
DocImpact
Implements blueprint xenapi-ipxe-iso-boot-support
Change-Id: I33acf9dfdff0a5ed9797723a142bc451348e8549
Send identity headers while uploading/downloading images from glance. Related
to:
1) Add identity headers while calling glanceclient from nova
Ife02059abbdce7920f1f408b71e9745d777fa770
2) Pass identity headers received in glanceclient to nova
Ifbef582aa4e64a2e7a46db43a9cc6cf8c3531dbd
3) Pass identity headers received by glance api to glance registry
Ie5f07ed6dfeaa8428de4f79c4d40d182328e6ab4
NOTE: since this is a change to a Dom0 plugin API, it requires the lock-step
upgrade of both the compute-manager and the dom0 plugin.
DocImpact
Change-Id: I6d5e3448d2c0acc392d18e0b88cec25cf313da5b
Recently a change was made to how/where the '_' method is imported. This
seems to have broken most xenserver plugins. These plugins shouldn't be
doing * imports anyway so I've gone in and cleaned up all places where
"from pluginlib_nova import *" was used and replaced with "import
pluginlib_nova".
Fixes bug 1207107
Change-Id: I3b2a299cf60543e0f766becfeb4740af791b3576
There are a few upload errors that glance plugin should
not retry upload on, as it will fail again. This patch changes
upload to not retry on the following conditions in addition
to the 401 error that it already handles:
- CONFLICT
- REQUEST_ENTITY_TOO_LARGE
- PRECONDITION_FAILED
- FORBIDDEN
Fixes bug 1203726
Change-Id: I538a4f9b133169b2e398e7dfcb3af58fef204f03
Previous _ was monkey patched into builtins whenever
certain modules were imported. This removes that and
simply imports it when it is needed.
Change-Id: I0af2c6d8a230e94440d655d13cab9107ac20d13c
It is possible for a token to expire during a very long image upload.
If this happens, there is little point in retrying the upload, as it
is most likely to fail again for the same reason.
Fixes bug 1199454
Change-Id: Id699135a8ec4c603671b16f0127d0b2cd2a55855
The current BitTorrent code was passing integer arguments to `make_subprocess`
which would in turn pass those to `popen`. This would cause a failure because
the semantics of the underlying `execve` function is to only accept a list of
strings.
The solution is to cast the arguments to a string before passing them into
`make_subrpocess`.
Fixes bug 1201594
Change-Id: I8372795ca032420641c449cb07bb47f9252ff4ef
Current the xenapi glance plugin does not close
httplib connections when exceptions are raised.
Note this code is run under python 2.4 so with
is not available.
Fixes bug 1197846
Change-Id: I5cfb067320e7506109c301532a95fe83aa4fd0f9
* Rewrite xenhost._run_command and xenstore._run_command to use
utils.make_subprocess and utils.finish_process
* Change exception raised by utils.finish_process to retain information
needed by calls in xenhost and xenstore
Change-Id: Idcdb50bededf0acde92f1774d6752043ba8f97ce
Fixes: bug #1074087
Change xenapi bittorrent plugin to not assume that torrent file
is under CONF.xenapi_torrent_base_url + image_id + ".torrent".
This allows for sources images from arbitrary torrent URLs.
Change-Id: Ica9462a51c6299b5affc861781f3054d18866458
Catch invalid-cross device linking exceptions raise from os.rename
and provide a more intelligible error message.
Change-Id: I22b6a145141431b6d4e4cb0176881f0cf0dd609f
If an entry is already gone from xenstore, any other xenstore-rm will
raise.
When we make calls to nova-agent, we wait for the response in Xenstore.
After a timeout period, we attempt to remove it. If the DomID has
changed (due to a Windows VM rebooting at the right time via sysprep,
etc)... or if the agent picked up the command and didn't respond in a
timely fashion, the removal attempt will fail. It would be normal for
the entry to be gone in this case.
This fixes the plugin to ignore when the entry is already
deleted.
Fixes bug 1193720
Change-Id: I8585a9f147cad3c5abe531620872b475eb4128c9
If an administrator has enabled the logging of guest
consoles on XCP/XenServer, this enables nova to return
the last MB of those logs to the user.
The management of the logs on the server is a little
tricky, and will be sorted in a later patch.
Change was based on this previous idea:
https://review.openstack.org/#/c/17959/
DocImpact
Part of blueprint xenapi-server-log
Change-Id: I23c83bcf8c648cc2714a0c78951acc29a16d5c31
* Handles the output of vhd-util check a bit more intelligently.
* Changes it to use LOG like in other modules
* Adds a necessary import (errno)
Change-Id: I75d6b9e081159bc29732a721deb7a557e5f2ee06
Fix for bug 1160323.
DocImpact
Depends on a version of XAPI supporting relax-xsm-sr-check.
Currently no release versions support this, so the change at
https://github.com/xen-org/xen-api/pull/1116 would need to be applied
to the source to enable this. XenServer 6.2 is expected to
support this mode, and possibly some future hotfixes for XenServer 6.1.
It also depends on a configuration change which must be documented and added to devstack:
* Set "relax-xsm-sr-check = true" in /etc/xapi.conf
This commit makes the following changes:
* Attach the SR on the destination host prior to migrate
* Returns the SR-ref from the pre_migrate call (API change)
* Populates the XS VDI map with the sr-ref on the destination host
* Removes the connection to the SR from the source host post-migrate
* Added plugin to determine if iSCSI block migration is enabled
Associated tempest test at https://review.openstack.org/#/c/26478/
Change-Id: I917d9cf094190d636f4b9e14f6c8e728ff85af0e
Fixes: bug 1160323
Fixes bug 1157695
This fix removes the parted dependency from the rpm specification file,
and also gets rid of the assumption, that the build-rpm.sh is executed
at a specific directory.
Change-Id: I1dcf9096a783a88bc09af0f4f4d5541a7e505748
Jenkins