Use force=True for os-brick disconnect during delete
The 'force' parameter of os-brick's disconnect_volume() method allows callers to ignore flushing errors and ensure that devices are being removed from the host. We should use force=True when we are going to delete an instance to avoid leaving leftover devices connected to the compute host which could then potentially be reused to map to volumes to an instance that should not have access to those volumes. We can use force=True even when disconnecting a volume that will not be deleted on termination because os-brick will always attempt to flush and disconnect gracefully before forcefully removing devices. Closes-Bug: #2004555 Change-Id: I3629b84d3255a8fe9d8a7cea8c6131d7c40899e8
This commit is contained in:
melanie witt
@@ -284,7 +284,7 @@ Troubleshooting
|
||||
Timeouts
|
||||
~~~~~~~~
|
||||
|
||||
Configure a :ref:`service user <user_token_timeout>` in case the user token
|
||||
Configure a :ref:`service user <service_user_token>` in case the user token
|
||||
times out, e.g. during the snapshot and download of a large server image.
|
||||
|
||||
If RPC calls are timing out with a ``MessagingTimeout`` error in the logs,
|
||||
|
||||
@@ -19,6 +19,7 @@ A list of config options based on different topics can be found below:
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
/admin/configuration/service-user-token
|
||||
/admin/configuration/api
|
||||
/admin/configuration/resize
|
||||
/admin/configuration/cross-cell-resize
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
.. _service_user_token:
|
||||
|
||||
===================
|
||||
Service User Tokens
|
||||
===================
|
||||
|
||||
.. note::
|
||||
|
||||
Configuration of service user tokens is **required** for every Nova service
|
||||
for security reasons. See https://bugs.launchpad.net/nova/+bug/2004555 for
|
||||
details.
|
||||
|
||||
Configure Nova to send service user tokens alongside regular user tokens when
|
||||
making REST API calls to other services. The identity service (Keystone) will
|
||||
authenticate a request using the service user token if the regular user token
|
||||
has expired.
|
||||
|
||||
This is important when long-running operations such as live migration or
|
||||
snapshot take long enough to exceed the expiry of the user token. Without the
|
||||
service token, if a long-running operation exceeds the expiry of the user
|
||||
token, post operations such as cleanup after a live migration could fail when
|
||||
Nova calls other service APIs like block-storage (Cinder) or networking
|
||||
(Neutron).
|
||||
|
||||
The service token is also used by services to validate whether the API caller
|
||||
is a service. Some service APIs are restricted to service users only.
|
||||
|
||||
To set up service tokens, create a ``nova`` service user and ``service`` role
|
||||
in the identity service (Keystone) and assign the ``service`` role to the
|
||||
``nova`` service user.
|
||||
|
||||
Then, configure the :oslo.config:group:`service_user` section of the Nova
|
||||
configuration file, for example:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = true
|
||||
auth_url = https://104.130.216.102/identity
|
||||
auth_strategy = keystone
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
username = nova
|
||||
password = secretservice
|
||||
...
|
||||
|
||||
And configure the other identity options as necessary for the service user,
|
||||
much like you would configure nova to work with the image service (Glance) or
|
||||
networking service (Neutron).
|
||||
|
||||
.. note::
|
||||
|
||||
Please note that the role assigned to the :oslo.config:group:`service_user`
|
||||
needs to be in the configured
|
||||
:oslo.config:option:`keystone_authtoken.service_token_roles` of other
|
||||
services such as block-storage (Cinder), image (Glance), and networking
|
||||
(Neutron).
|
||||
@@ -320,4 +320,4 @@ To make live-migration succeed, you have several options:
|
||||
|
||||
If live migrations routinely timeout or fail during cleanup operations due
|
||||
to the user token timing out, consider configuring nova to use
|
||||
:ref:`service user tokens <user_token_timeout>`.
|
||||
:ref:`service user tokens <service_user_token>`.
|
||||
|
||||
@@ -67,7 +67,7 @@ Create a snapshot of the instance
|
||||
|
||||
If snapshot operations routinely fail because the user token times out
|
||||
while uploading a large disk image, consider configuring nova to use
|
||||
:ref:`service user tokens <user_token_timeout>`.
|
||||
:ref:`service user tokens <service_user_token>`.
|
||||
|
||||
#. Use the :command:`openstack image list` command to check the status
|
||||
until the status is ``ACTIVE``:
|
||||
|
||||
@@ -478,67 +478,3 @@ Ensure the ``compute`` endpoint in the identity service catalog is pointing
|
||||
at ``/v2.1`` instead of ``/v2``. The former route supports microversions,
|
||||
while the latter route is considered the legacy v2.0 compatibility-mode
|
||||
route which renders all requests as if they were made on the legacy v2.0 API.
|
||||
|
||||
|
||||
.. _user_token_timeout:
|
||||
|
||||
User token times out during long-running operations
|
||||
---------------------------------------------------
|
||||
|
||||
Problem
|
||||
~~~~~~~
|
||||
|
||||
Long-running operations such as live migration or snapshot can sometimes
|
||||
overrun the expiry of the user token. In such cases, post operations such
|
||||
as cleaning up after a live migration can fail when the nova-compute service
|
||||
needs to cleanup resources in other services, such as in the block-storage
|
||||
(cinder) or networking (neutron) services.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec
|
||||
2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled
|
||||
2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client
|
||||
2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance
|
||||
2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
|
||||
|
||||
Solution
|
||||
~~~~~~~~
|
||||
|
||||
Configure nova to use service user tokens to supplement the regular user token
|
||||
used to initiate the operation. The identity service (keystone) will then
|
||||
authenticate a request using the service user token if the user token has
|
||||
already expired.
|
||||
|
||||
To use, create a service user in the identity service similar as you would when
|
||||
creating the ``nova`` service user.
|
||||
|
||||
Then configure the :oslo.config:group:`service_user` section of the nova
|
||||
configuration file, for example:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = True
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
password = secretservice
|
||||
username = nova
|
||||
auth_url = https://104.130.216.102/identity
|
||||
...
|
||||
|
||||
And configure the other identity options as necessary for the service user,
|
||||
much like you would configure nova to work with the image service (glance)
|
||||
or networking service.
|
||||
|
||||
.. note::
|
||||
|
||||
Please note that the role of the :oslo.config:group:`service_user` you
|
||||
configure needs to be a superset of
|
||||
:oslo.config:option:`keystone_authtoken.service_token_roles` (The option
|
||||
:oslo.config:option:`keystone_authtoken.service_token_roles` is configured
|
||||
in cinder, glance and neutron).
|
||||
|
||||
@@ -92,6 +92,26 @@ Install and configure components
|
||||
Comment out or remove any other options in the ``[keystone_authtoken]``
|
||||
section.
|
||||
|
||||
* In the ``[service_user]`` section, configure :ref:`service user
|
||||
tokens <service_user_token>`:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = true
|
||||
auth_url = https://controller/identity
|
||||
auth_strategy = keystone
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
username = nova
|
||||
password = NOVA_PASS
|
||||
|
||||
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
|
||||
the Identity service.
|
||||
|
||||
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
|
||||
@@ -84,6 +84,26 @@ Install and configure components
|
||||
Comment out or remove any other options in the ``[keystone_authtoken]``
|
||||
section.
|
||||
|
||||
* In the ``[service_user]`` section, configure :ref:`service user
|
||||
tokens <service_user_token>`:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = true
|
||||
auth_url = https://controller/identity
|
||||
auth_strategy = keystone
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
username = nova
|
||||
password = NOVA_PASS
|
||||
|
||||
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
|
||||
the Identity service.
|
||||
|
||||
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
|
||||
@@ -74,6 +74,26 @@ Install and configure components
|
||||
Comment out or remove any other options in the
|
||||
``[keystone_authtoken]`` section.
|
||||
|
||||
* In the ``[service_user]`` section, configure :ref:`service user
|
||||
tokens <service_user_token>`:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = true
|
||||
auth_url = https://controller/identity
|
||||
auth_strategy = keystone
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
username = nova
|
||||
password = NOVA_PASS
|
||||
|
||||
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
|
||||
the Identity service.
|
||||
|
||||
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
|
||||
@@ -260,6 +260,26 @@ Install and configure components
|
||||
Comment out or remove any other options in the ``[keystone_authtoken]``
|
||||
section.
|
||||
|
||||
* In the ``[service_user]`` section, configure :ref:`service user
|
||||
tokens <service_user_token>`:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = true
|
||||
auth_url = https://controller/identity
|
||||
auth_strategy = keystone
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
username = nova
|
||||
password = NOVA_PASS
|
||||
|
||||
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
|
||||
the Identity service.
|
||||
|
||||
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the
|
||||
management interface IP address of the controller node:
|
||||
|
||||
|
||||
@@ -247,6 +247,26 @@ Install and configure components
|
||||
Comment out or remove any other options in the ``[keystone_authtoken]``
|
||||
section.
|
||||
|
||||
* In the ``[service_user]`` section, configure :ref:`service user
|
||||
tokens <service_user_token>`:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = true
|
||||
auth_url = https://controller/identity
|
||||
auth_strategy = keystone
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
username = nova
|
||||
password = NOVA_PASS
|
||||
|
||||
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
|
||||
the Identity service.
|
||||
|
||||
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the
|
||||
management interface IP address of the controller node:
|
||||
|
||||
|
||||
@@ -237,6 +237,26 @@ Install and configure components
|
||||
Comment out or remove any other options in the ``[keystone_authtoken]``
|
||||
section.
|
||||
|
||||
* In the ``[service_user]`` section, configure :ref:`service user
|
||||
tokens <service_user_token>`:
|
||||
|
||||
.. path /etc/nova/nova.conf
|
||||
.. code-block:: ini
|
||||
|
||||
[service_user]
|
||||
send_service_user_token = true
|
||||
auth_url = https://controller/identity
|
||||
auth_strategy = keystone
|
||||
auth_type = password
|
||||
project_domain_name = Default
|
||||
project_name = service
|
||||
user_domain_name = Default
|
||||
username = nova
|
||||
password = NOVA_PASS
|
||||
|
||||
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
|
||||
the Identity service.
|
||||
|
||||
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the
|
||||
management interface IP address of the controller node:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user