Compute manager to use thread pools selectively

This changes the thread pool usage of the ComputeManager to go through
the concurrency mode aware util functions.

The concurrent live migration pool had a seemingly unlimited option
when configured with value 0, but in reality GreenThreadPool has a
default worker size of 1000. In reality it is almost never right to
have more than one live migration running concurrently. Also with
native threading having 1000 worker is just too costly. So we
decided to deprecate the value 0 and changed the implementation of
unlimited to mean 5 threads in native threading mode. We kept the 1000
greenthread in eventlet mode for backward compatibility.

The _sync_power_states periodic task also spawn tasks for each instance
to be synced. As it uses a shared data structure across these tasks
and the caller a lock is needed to avoid race conditions.
Also the default pool size is 1000 for these tasks in our configuration.
That would use a lot of memory on a busy host in native threading mode.
So we changed the default value from 1000 to 5.

Change-Id: I9567d5fabdf086b5d0493103d9f6bde4f66af387
Signed-off-by: Balazs Gibizer <gibi@redhat.com>

releasenotes/notes/deprecate-unlimited-max_concurrent_live_migrations-29c54c7eeb77041c.yaml

@@ -0,0 +1,28 @@
+---
+upgrade:
+  - |
+    The meaning of the 0 value of the config option
+    ``[DEFAULT]max_concurrent_live_migrations`` has been changed. In the past
+    the implementation of the meaning of "unlimited" used maximum 1000
+    concurrent worker greenthreads. For eventlet mode this behavior is kept but
+    for the native threading mode it is now reduced to 5 native threads. It is
+    almost always a bad idea to change this config option from its default value, 1.
+    Please read the `concurrency
+    <https://docs.openstack.org/nova/latest/admin/concurrency.html>`__
+    guide for more details.
+  - |
+    The default value of the configuration option
+    ``[DEFAULT]sync_power_state_thread_pool_size`` is changed from 1000 to 5 to
+    have a value that is safe to use in native threading mode. If you are still
+    using the eventlet mode and relying on a higher value then configure that
+    higher value explicitly before the upgrade. Please read the
+    `concurrency <https://docs.openstack.org/nova/latest/admin/concurrency.html>`__
+    guide for more details.
+deprecations:
+  - |
+    The possible 0 value of the configuration option
+    ``[DEFAULT]max_concurrent_live_migrations`` is deprecated and will be
+    removed in a future release. It is almost always a bad idea to change the
+    default value, 1, of this config option. If more performant live migration
+    is needed, use the ``live_migration_parallel_connections`` config option
+    instead.