Ghanshyam Maan
d5ffb58a8d
For graceful shutdown of compute service, it will have two RPC servers.
One RPC server is used for the new requests which will be stopped during
graceful shutdown and 2nd RPC server (listen on 'compute-alt' topic)
will be used to complete the in-progress operations.
We select the operations (case by case) and their RPC method to use
the 2nd PRC server so that they will not be interupted on shutdown
initiative and graceful shutdown time will keep 2nd RPC server active
for graceful_shutdown_timeout. A new method 'prepare_for_alt_rpcserver'
is added which will fallback to first RPC server if it detect the old
compute.
As this is upgrade impact, it bumps the compute/service version, adds
releasenotes for the same.
The list of operations who should use the 2nd RPC server will grow
evanutally and this commit moves the below operations to use the 2nd
RPC server:
* Live migration
- Live migration: It use 2nd RPC servers and will try to complete
the operation during shutdown.
- live_migration_force_complete does not need to use 2nd RPC server.
It is direct RPC request from API to compute and if that is
rejected during shutdown, it is fine and can be initiated again
once compute is up.
- live_migration_abort does not need to use 2nd RPC server. Ditto,
it is direct RPC request from API to compute. It cancel the queue
live migration but if migration is already started, then driver
cancel the migration. If it is rejected during shutdown because of
RPC is stopped, it is fine and can be initiated again.
* server external event
* Get server console
As graceful shutdown cannot be tested in tempest, this adds a new job
to test it. Currently it test the live migration operation which can
be extended to other operations who will use 2nd RPC server.
Partial implement blueprint nova-services-graceful-shutdown-part1
Change-Id: I4de3afbcfaefbed909a29a831ac18060c4a73246
Signed-off-by: Ghanshyam Maan <gmaan.os14@gmail.com>
57 lines
2.3 KiB
YAML
57 lines
2.3 KiB
YAML
- name: Graceful shutdown source compute live migration
|
|
block:
|
|
- name: Start live migrations of test servers
|
|
become: true
|
|
become_user: stack
|
|
script: "start_live_migration.sh server-lm1"
|
|
environment:
|
|
SUBNODE_HOSTNAME: "{{ hostvars['compute1']['ansible_hostname'] }}"
|
|
CONTROLLER_HOSTNAME: "{{ hostvars['controller']['ansible_hostname'] }}"
|
|
register: start_live_migrations_result
|
|
failed_when: start_live_migrations_result.rc not in [0, 2]
|
|
|
|
- name: Set fact if migrations completed or timed out before SIGTERM to source compute
|
|
set_fact:
|
|
live_migrations_completed_or_timeout: "{{ start_live_migrations_result.rc == 2 }}"
|
|
|
|
- name: Run graceful shutdown tests
|
|
when: not live_migrations_completed_or_timeout
|
|
block:
|
|
- name: Send SIGTERM to source compute to start the source compute graceful shutdown
|
|
delegate_to: compute1
|
|
become: true
|
|
shell: "kill -15 $(systemctl show devstack@n-cpu -p MainPID --value)"
|
|
|
|
- name: Verify live migration is completed during graceful shutdown
|
|
become: true
|
|
become_user: stack
|
|
script: "verify_live_migration.sh server-lm1"
|
|
environment:
|
|
CONTROLLER_HOSTNAME: "{{ hostvars['controller']['ansible_hostname'] }}"
|
|
|
|
# Sleep for 180 sec: default graceful_shutdown_timeout
|
|
- name: Sleep for 180 seconds to allow source compute graceful shutdown to complete
|
|
pause:
|
|
seconds: 180
|
|
|
|
- name: Verify compute service is stopped after graceful shutdown
|
|
become: true
|
|
become_user: stack
|
|
script: "start_and_verify_compute_service.sh {{ hostvars['compute1']['ansible_hostname'] }} inactive"
|
|
|
|
- name: Start and verify subnode compute service is running
|
|
become: true
|
|
become_user: stack
|
|
script: "start_and_verify_compute_service.sh {{ hostvars['compute1']['ansible_hostname'] }}"
|
|
|
|
- name: Cleanup test servers
|
|
become: true
|
|
become_user: stack
|
|
script: "cleanup_test_servers.sh server-lm1"
|
|
ignore_errors: true
|
|
|
|
- name: Fail if any test is skipped
|
|
fail:
|
|
msg: "One or more test is skipped due to operation is either completed or timed out before SIGTERM signal."
|
|
when: live_migrations_completed_or_timeout
|