From 6fa5bd43fb0ea41d26a5cc1996ce355a14ce0d5b Mon Sep 17 00:00:00 2001 From: Chris Dent Date: Thu, 15 Aug 2019 11:52:19 +0100 Subject: [PATCH] Add a "Caveats" section to the eventlet profiling docs There are some bits of wisdom and workarounds required to have a positive experience when profiling in an eventlet environment. This patch adds a section where such wisdom can accumlate, and also provides a workaround for a specific problem when profiling nova-compute. Change-Id: Id6362f20c831c43e4d3316fe573e28c6b891d459 --- .../testing/eventlet-profiling.rst | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/doc/source/contributor/testing/eventlet-profiling.rst b/doc/source/contributor/testing/eventlet-profiling.rst index b89ab16149..7cc389c636 100644 --- a/doc/source/contributor/testing/eventlet-profiling.rst +++ b/doc/source/contributor/testing/eventlet-profiling.rst @@ -197,6 +197,9 @@ Change to that directory and view the profile using the pstats python3 -m pstats ef044142-f3b8-409d-9af6-c60cea39b273.prof +.. note:: The major version of python used to analyze the profile data must be + the same as the version used to run the process being profiled. + Sort stats by their cumulative time:: ef044142-f3b8-409d-9af6-c60cea39b273.prof% sort cumtime @@ -232,6 +235,79 @@ From this we can make a couple of useful inferences about ``get_by_host``: Several other sort modes can be used. List those that are available by entering ``sort`` without arguments. +Caveats +======= + +Real world use indicates that the eventlet profiler is not perfect. There are +situations where it will not always track switches between greenlets as well as +it could. This can result in profile data that does not make sense or random +slowdowns in the system being profiled. There is no one size fits all solution +to these issues; profiling eventlet services is more an art than science. +However, this section tries to provide a (hopefully) growing body of advice on +what to do to work around problems. + +General Advice +-------------- + +* Try to profile chunks of code that operate mostly within one module or class + and do not have many collaborators. The more convoluted the path through + the code, the more confused the profiler gets. + +* Similarly, where possible avoid profiling code that will trigger many + greenlet context switches; either specific spawns, or multiple types of I/O. + Instead, narrow the focus of the profiler. + +* If possible, avoid RPC. + +In nova-compute +--------------- + +The creation of this caveat section was inspired by issues experienced while +profiling ``nova-compute``. The ``nova-compute`` process is not allowed to +speak with a database server directly. Instead communication is mediated +through the conductor, communication happening via ``oslo.versionedobjects`` +and remote calls. Profiling methods such as ``_update_available_resources`` in +the ResourceTracker, which needs information from the database, results in +profile data that can be analyzed but is incorrect and misleading. + +This can be worked around by temporarily changing ``nova-compute`` to allow it +to speak to the database directly: + +.. code-block:: diff + + diff --git a/nova/cmd/compute.py b/nova/cmd/compute.py + index 01fd20de2e..655d503158 100644 + --- a/nova/cmd/compute.py + +++ b/nova/cmd/compute.py + @@ -50,8 +50,10 @@ def main(): + + gmr.TextGuruMeditation.setup_autorun(version, conf=CONF) + + - cmd_common.block_db_access('nova-compute') + - objects_base.NovaObject.indirection_api = conductor_rpcapi.ConductorAPI() + + # Temporarily allow access to the database. You must update the config file + + # used by this process to set [database]/connection to the cell1 database. + + # cmd_common.block_db_access('nova-compute') + + # objects_base.NovaObject.indirection_api = conductor_rpcapi.ConductorAPI() + objects.Service.enable_min_version_cache() + server = service.Service.create(binary='nova-compute', + topic=compute_rpcapi.RPC_TOPIC) + +The configuration file used by the ``nova-compute`` process must also be +updated to ensure that it contains a setting for the relevant database: + +.. code-block:: ini + + [database] + connection = mysql+pymysql://root:secret@127.0.0.1/nova_cell1?charset=utf8 + +In a single node devstack setup ``nova_cell1`` is the right choice. The +connection string will vary in other setups. + +Once these changes are made, along with the profiler changes indicated in the +example above, ``nova-compute`` can be restarted and with luck some useful +profiling data will emerge. + .. _eventlet: https://eventlet.net/ .. _cProfile: https://docs.python.org/3/library/profile.html .. _SnakeViz: https://jiffyclub.github.io/snakeviz/