From f9e13bcfed107c1b91b98c72e72ebbaba7b979ee Mon Sep 17 00:00:00 2001 From: Eric Fried Date: Thu, 15 Mar 2018 11:40:09 -0500 Subject: [PATCH] update_provider_tree devref and docstring updates Design changes [1] from the Dublin PTG prompted some rewording of the docstring for ComputeDriver.update_provider_tree. And to avoid that docstring becoming too enormous, relevant chunks of the spec [2] are copied to a new devref document which is linked from that docstring. [1] https://review.openstack.org/#/c/552122/ [2] http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/update-provider-tree.html Change-Id: I06504aa2a3fe6d39ecc1e681de43be8fee9e06f6 blueprint: update-provider-tree --- doc/source/index.rst | 1 + doc/source/reference/index.rst | 2 + doc/source/reference/update-provider-tree.rst | 172 ++++++++++++++++++ nova/virt/driver.py | 42 +++-- 4 files changed, 202 insertions(+), 15 deletions(-) create mode 100644 doc/source/reference/update-provider-tree.rst diff --git a/doc/source/index.rst b/doc/source/index.rst index bc5ae6708e..5600286dc9 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -266,6 +266,7 @@ looking parts of our architecture. These are collected below. reference/services reference/stable-api reference/threading + reference/update-provider-tree reference/vm-states user/index user/aggregates diff --git a/doc/source/reference/index.rst b/doc/source/reference/index.rst index 3c66005e09..b763b3732c 100644 --- a/doc/source/reference/index.rst +++ b/doc/source/reference/index.rst @@ -22,6 +22,8 @@ The following is a dive into some of the internals in nova. based on eventlet, and may not be familiar to everyone. * :doc:`/reference/notifications`: How the notifications subsystem works in nova, and considerations when adding notifications. +* :doc:`/reference/update-provider-tree`: A detailed explanation of the + ``ComputeDriver.update_provider_tree`` method. Debugging ========= diff --git a/doc/source/reference/update-provider-tree.rst b/doc/source/reference/update-provider-tree.rst new file mode 100644 index 0000000000..e165f61104 --- /dev/null +++ b/doc/source/reference/update-provider-tree.rst @@ -0,0 +1,172 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +==================================== + ComputeDriver.update_provider_tree +==================================== + +This provides details on the ``ComputeDriver`` abstract method +``update_provider_tree`` for developers implementing this method in their own +virt drivers. + +Background +---------- +In the movement towards using placement for scheduling and resource management, +the virt driver method ``get_available_resource`` was initially superseded by +``get_inventory``, whereby the driver could specify its inventory in terms +understood by placement. In Queens, a ``get_traits`` driver method was added. +But ``get_inventory`` is limited to expressing only inventory (not traits or +aggregates). And both of these methods are limited to the resource provider +corresponding to the compute node. + +Recent developments such as Nested Resource Providers necessitate the ability +for the virt driver to have deeper control over what the resource tracker +configures in placement on behalf of the compute node. This need is filled by +the virt driver method ``update_provider_tree`` and its consumption by the +resource tracker, allowing full control over the placement representation of +the compute node and its associated providers. + +The Method +---------- +``update_provider_tree`` accepts two parameters: + +* A ``nova.compute.provider_tree.ProviderTree`` object representing all the + providers in the tree associated with the compute node, and any sharing + providers (those with the ``MISC_SHARES_VIA_AGGREGATE`` trait) associated via + aggregate with any of those providers (but not *their* tree- or + aggregate-associated providers), as currently known by placement. This + object is fully owned by the ``update_provider_tree`` method, and can + therefore be modified without locking/concurrency considerations. In other + words, the parameter is passed *by reference* with the expectation that the + virt driver will modify the object. Note, however, that it may contain + providers not directly owned/controlled by the compute host. Care must be + taken not to remove or modify such providers inadvertently. In addition, + providers may be associated with traits and/or aggregates maintained by + outside agents. The ``update_provider_tree`` method must therefore also be + careful only to add/remove traits/aggregates it explicitly controls. +* String name of the compute node (i.e. ``ComputeNode.hypervisor_hostname``) + for which the caller is requesting updated provider information. Drivers may + use this to help identify the compute node provider in the ProviderTree. + Drivers managing more than one node (e.g. ironic) may also use it as a cue to + indicate which node is being processed by the caller. + +The virt driver is expected to update the ProviderTree object with current +resource provider and inventory information. When the method returns, the +ProviderTree should represent the correct hierarchy of nested resource +providers associated with this compute node, as well as the inventory, +aggregates, and traits associated with those resource providers. + +.. note:: Despite the name, a ProviderTree instance may in fact contain more + than one tree. For purposes of this specification, the ProviderTree + passed to ``update_provider_tree`` will contain: + + * the entire tree associated with the compute node; and + * any sharing providers (those with the ``MISC_SHARES_VIA_AGGREGATE`` + trait) which are associated via aggregate with any of the providers + in the compute node's tree. The sharing providers will be + presented as lone roots in the ProviderTree, even if they happen to + be part of a tree themselves. + + Consider the example below. ``SSP`` is a shared storage provider and + ``BW1`` and ``BW2`` are shared bandwidth providers; all three have + the ``MISC_SHARES_VIA_AGGREGATE`` trait:: + + CN1 SHR_ROOT CN2 + / \ agg1 / /\ agg1 / \ + NUMA1 NUMA2--------SSP--/--\-----------NUMA1 NUMA2 + / \ / \ / \ / \ / \ + PF1 PF2 PF3 PF4--------BW1 BW2------PF1 PF2 PF3 PF4 + agg2 agg3 + + When ``update_provider_tree`` is invoked for ``CN1``, it is passed a + ProviderTree containing:: + + CN1 (root) + / \ agg1 + NUMA1 NUMA2-------SSP (root) + / \ / \ + PF1 PF2 PF3 PF4------BW1 (root) + agg2 + +This method supersedes ``get_inventory`` and ``get_traits``: if this method is +implemented, neither ``get_inventory`` nor ``get_traits`` is used. + +Driver implementations of ``update_provider_tree`` are expected to use public +``ProviderTree`` methods to effect changes to the provider tree passed in. +Some of the methods which may be useful are as follows: + +* ``new_root``: Add a new root provider to the tree. +* ``new_child``: Add a new child under an existing provider. +* ``data``: Access information (name, UUID, parent, inventory, traits, + aggregates) about a provider in the tree. +* ``remove``: Remove a provider **and its descendants** from the tree. Use + caution in multiple-ownership scenarios. +* ``update_inventory``: Set the inventory for a provider. +* ``add_traits``, ``remove_traits``: Set/unset virt-owned traits for a + provider. +* ``add_aggregates``, ``remove_aggregates``: Set/unset virt-owned aggregate + associations for a provider. + +.. note:: There is no supported mechanism for ``update_provider_tree`` to + effect changes to allocations. This is intentional: in Nova, + allocations are managed exclusively outside of virt. (Usually by the + scheduler; sometimes - e.g. for migrations - by the conductor.) + +Porting from get_inventory +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Virt driver developers wishing to move from ``get_inventory`` to +``update_provider_tree`` should use the ``ProviderTree.update_inventory`` +method, specifying the compute node as the provider and the same inventory as +returned by ``get_inventory``. For example: + +.. code:: + + def get_inventory(self, nodename): + inv_data = { + 'VCPU': { ... }, + 'MEMORY_MB': { ... }, + 'DISK_GB': { ... }, + } + return inv_data + +would become: + +.. code:: + + def update_provider_tree(self, provider_tree, nodename): + inv_data = { + 'VCPU': { ... }, + 'MEMORY_MB': { ... }, + 'DISK_GB': { ... }, + } + provider_tree.update_inventory(nodename, inv_data) + +Porting from get_traits +~~~~~~~~~~~~~~~~~~~~~~~ +To replace ``get_traits``, developers should use the +``ProviderTree.add_traits`` method, specifying the compute node as the +provider and the same traits as returned by ``get_traits``. For example: + +.. code:: + + def get_traits(self, nodename): + traits = ['HW_CPU_X86_AVX', 'HW_CPU_X86_AVX2', 'CUSTOM_GOLD'] + return traits + +would become: + +.. code:: + + def update_provider_tree(self, provider_tree, nodename): + provider_tree.add_traits( + nodename, 'HW_CPU_X86_AVX', 'HW_CPU_X86_AVX2', 'CUSTOM_GOLD') diff --git a/nova/virt/driver.py b/nova/virt/driver.py index a06e960716..07bf028dba 100644 --- a/nova/virt/driver.py +++ b/nova/virt/driver.py @@ -845,23 +845,35 @@ class ComputeDriver(object): :note: Renaming a provider (by deleting it from provider_tree and re-adding it with a different name) is not supported at this time. + See the developer reference documentation for more details: + + https://docs.openstack.org/nova/latest/reference/update-provider-tree.html # noqa + :param nova.compute.provider_tree.ProviderTree provider_tree: - A ProviderTree object representing all the providers associated - with the compute node, and any sharing providers (those with the - ``MISC_SHARES_VIA_AGGREGATE`` trait) associated via aggregate with - any of those providers (but not *their* tree- or aggregate- - associated providers), as currently known by placement. This - object is fully owned by the ``update_provider_tree`` method, and - can therefore be modified without locking/concurrency - considerations. Note, however, that it may contain providers not - directly owned/controlled by the compute host. Care must be taken - not to remove or modify such providers inadvertently. + A nova.compute.provider_tree.ProviderTree object representing all + the providers in the tree associated with the compute node, and any + sharing providers (those with the ``MISC_SHARES_VIA_AGGREGATE`` + trait) associated via aggregate with any of those providers (but + not *their* tree- or aggregate-associated providers), as currently + known by placement. This object is fully owned by the + update_provider_tree method, and can therefore be modified without + locking/concurrency considerations. In other words, the parameter + is passed *by reference* with the expectation that the virt driver + will modify the object. Note, however, that it may contain + providers not directly owned/controlled by the compute host. Care + must be taken not to remove or modify such providers inadvertently. + In addition, providers may be associated with traits and/or + aggregates maintained by outside agents. The + `update_provider_tree`` method must therefore also be careful only + to add/remove traits/aggregates it explicitly controls. :param nodename: - Name of the compute node for which the caller is updating providers - and inventory. Drivers managing more than one node may use this in - an advisory capacity to restrict changes to only the providers - associated with that one node, but this is not a requirement: the - caller always subsumes all changes regardless. + String name of the compute node (i.e. + ComputeNode.hypervisor_hostname) for which the caller is requesting + updated provider information. Drivers may use this to help identify + the compute node provider in the ProviderTree. Drivers managing + more than one node (e.g. ironic) may also use it as a cue to + indicate which node is being processed by the caller. + :return: True if the provider_tree was changed; False otherwise. """ raise NotImplementedError()