Since the introduction of quoting of header content in
https://review.openstack.org/568698, the 'x-image-meta-location' header
has been broken, because urllib.quote() is really intended to be applied
to only the path section of a URL, but in this case, it gets applied to
the entire URL, and catches the colon that separates the scheme from the
remainder of the URL.
This change adds the colon to the list of characters that should not get
quoted. Since a colon can be directly represented in ASCII, this should
not invalidate the previous change.
Change-Id: I76a1c9a361b6c9f6eb95ae766b8c3bcf2267703a
Closes-Bug: #1788942
If the token is encoded by base64, it may contain '=' char
as well.
We should skip quoting it.
Change-Id: I1ca63d251fa366f0e8e58128d45b729a2489b65c
Partial-Bug: #1783290
The token in request header may contain url char, such as '+',
if quote it, '+' will change to '%2B' which will lead to 401 error.
Our CI doesn't notice this bug because Keystone use fernet token
which doesn't contain url char by default. But token format in
keystone is plugable, some out-tree token formats may contain
url char (for example, PKI/PKIZ token).
So we should skip quote token to avoiding information changing.
Closes-bug: #1783290
Change-Id: I5aa71b3e2b9b19581e46ccf8a80eda5d637f17d1
According to RFC 8187, HTTP headers should use 7-bit ASCII encoding.
The glanceclient was encoding them as UTF-8, which can leave the 8th
bit nonzero when representing unicode, and which presents problems
for any recipient following the standard and decoding the headers as
ASCII.
This change requires keystoneauth1 3.6.2, which has a fix for a
bug that made it unable to handle bytes in headers. The dependency
is a patch bumping the keystoneauth1 version in upper-constraints.
Depends-on: https://review.openstack.org/#/c/569138/
Change-Id: I0d14974126fcb20e23a37347f4f1756c323cf2f5
Closes-bug: #1766235
--no-ssl-compression is deprecated and no longer used. So, it is
removed from the help message.
Change-Id: I2b886671a568ed191ee380cf16335ccd9ae85062
Closes-Bug: #1583919
The library is used in glanceclient/common/https.py and the
documentation build for the API fails without the dependency.
Update the error handling so that when OpenSSL reports an error it is
converted to a client communication error.
Change-Id: I0c0fb3139bb848d0cbaf88ae6a767a730bea74eb
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
This allows us to pass in a global_request_id in the client
constructor so that subsequent calls with this client pass that to the
servers. This enables cross project request_id tracking.
oslo spec I65de8261746b25d45e105394f4eeb95b9cb3bd42
Change-Id: Iea1e754a263a01dae5ed598fdda134394aff54b0
The bug: any existing Content-Type header cannot be found because the
call to headers.get() fails. Therefore we end up with two Content-Type
headers because a new one (applicaion/octet-stream) gets added
unconditionally. The cause: the strings (keys and values) in the headers
dict are converted from unicode sequences of type <str> to utf-8
sequences of type <bytes>. This happens in safe_encode()
(oslo_utils/encodeutils.py:66). <str> != <bytes> even if they appear to
have the same characters.
Hence, for python 3.x, _set_common_request_kwargs() adds content-type
to header even if custom content-type is set in the request.
This results in unsupported media type exception when glance client
is used with keystoneauth and python 3.x
The fix: follow the directions in encode_headers().
It says to do this just before sending the request. Honor this principle;
do not encode headers and then perform more business logic on them.
Change-Id: Idf6079b32f70bc171f5016467048e917d42f296d
Closes-bug: #1641239
Co-Authored-By: Pushkar Umaranikar <pushkar.umaranikar@intel.com>
This requests commit [1] changed the behavior when a nonexistent cacert
file is passed in: now it raises IOError. This is getting through
glanceclient.common.http.HTTPClient._request, which used to raise
CommunicationError in this scenario.
Even though there is arguably a better exception than CommunicationError
to represent this condition (like maybe IOError), for backward
compatibility this change set converts IOError to CommunicationError.
We also improve the unit test to raise the original exception if the
expected conditions aren't met; this improves debugability.
[1] https://github.com/kennethreitz/requests/commit/7d8b87c37f3a5fb993fd83eda6888ac217cd108e
Change-Id: I6a2cf4c6d041b67d3509153b4cef18b459263648
Closes-Bug: #1692085
In the recent release of keystoneauth1 2.18.0 provision is made to log
x-openstack-request-id for session client. Once this new library is synced
in openstack projects, the x-openstack-request-id will be logged twice
on the console if session client is used.
For example,
$ glance --debug image-list
DEBUG:keystoneauth.session:GET call to image for
http://10.232.48.204:9292/v2/schemas/image used request id
req-96a3f203-c605-4c96-a31d-f1199d41705c
DEBUG:glanceclient.common.http:GET call to glance-api for
http://10.232.48.204:9292/v2/schemas/image used request id
req-96a3f203-c605-4c96-a31d-f1199d41705c
Above log will be logged twice on the console.
Removed logging of x-openstack-request-id in case of SessionClient as it
is already logged in keystoneauth1. x-openstack-request-id will only be
logged once on console if HTTPClient is used.
Depends-On: I492b331ff3da8d0b91178bf0d5fe1d3702f15bd7
Closes-Bug: #1657351
Change-Id: I64258f997dc060113f53682aee74bdd40a346e54
1.As mentioned in [1], we should avoid usingg
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: I71c13040318eca6e5ed993e8aa03f8003986a71c
Refactoring code: Making dict to use single instruction: pop()
rather than two instructions: get() and del, giving the codes a
format that carries through.
TrivialRefactoring
Change-Id: Idb21df37c287fdff24c29153676f82544f735297
* exc.py: Encode body in response before calling replace over it.
* http.py: prepend the bytes literal to the empty string or else
we hit bug 1342080 again in python 3.
* utils.py: Use sys.stdin.buffer in python 3.
Change-Id: Ieefb8c633658e507486438e5518c5d53e819027d
As of now request-id is not logged if an excpetion
is raised.
Rearranged code so that request-id is logged even in case of
an exception.
Change-Id: Iee0398404ee752c0d880edf3054207c35862e71a
Closes-Bug: #1603863
String interpolation should be delayed to be handled by the logging
code, rather than being done at the point of the logging call.
See the oslo i18n guideline.
* http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Change-Id: If06663076e4081c6268ba88c157513723b734b31
Closes-Bug: #1596829
Added support to log 'X-Openstack-Request-Id' for each api call.
If glanceclient is used from command line then following log will
be logged on console if --debug flag is used.
DEBUG:glanceclient.common.http:GET call to glance-api for
http://172.26.88.20:9292/v2/schemas/image used request id
req-e0c7c97a-8fc0-4ce3-a669-d0b1eb5d7aae
If python-glanceclient is used in applications (e.g. Nova) then
following log message will be logged in service logs.
DEBUG glanceclient.common.http
[req-be074f1e-1c17-4786-b703-2a221751c8f4 demo demo] GET call to
glance-api for
http://172.26.88.20:9292/v1/images/detail?is_public=none&limit=20
used request id req-9b1dd929-df30-46b2-a8f2-dfd6ffbad3fc
DocImpact:
To use this feature user need to set 'default_log_levels' in third
party application. For example nova uses glance then in nova.conf
'default_log_levels' should be set as below:
default_log_levels = glanceclient=DEBUG
Implements: blueprint log-request-id
Change-Id: Ib04a07bac41ad2a5e997348f3b0bccc640169dc9
This reverts commit 9e532db8b0.
If glanceclient is used in multi-threaded environment, then there is a
possibility of getting invalid/wrong last request-id. To avoid this,
need to use thread local storage to store last-request-id and add
public method to return this request-id to caller.
http://specs.openstack.org/openstack/openstack-specs/specs/return-request-id.html#alternatives
Change-Id: I08d8d87fc0cc291f1b930b2c0cfc110ec8394131
Headers were encoded in HTTPClient, but when glance client started
to use SessionClient this functionality was lost.
This commit replaces static method "encode_headers" from HTTPClient
and makes it a common function, that SessionClient can use when
converting image meta to headers.
Change-Id: If9f8020220d2a0431b4241b38b9c83c09c0d75cb
Closes-bug: #1574587
apiclient.base.Resource.get method requires manager.client to have
last_request_id member. Otherwise get operation fails with
AttributeError exception.
Change-Id: I0ece85e3f61f2a7f176520ddf3ebee7792e51993
Closes-bug: 1552533
Previously auth_token was initialized once in __init__ method.
After that we stored token in session headers. So to refresh token
users need to instantiate a new session inside http client or
re-create client itself.
In order to provide possibility to refresh token we need
to add token header before sending the request. So users can
just update auth_token attribute in the HTTPClient to refresh
user token.
Change-Id: Ifebe9011870bbddc46fc6d6a26563641d5559e97
Closes-Bug: #1563495
glance has a code specific for python2.5. We need to delete this
code cause glanceclient doesn't support neither python2.5 or
python2.6.
Change-Id: I17e4905b6e02fcfff033a6cde03324e2a47bfce2
Not only is this code broken on the requests we require on
distro-provided requests, it's not needed anymore. Remove it.
Closes-bug: 1526254
Change-Id: I47a07bf9910f118392785fc20e015f036a2e8a7c
We need all the headers to be safe strings so they can be joined
together and not become an unicode string in doing so.
This fixes a bug when creating an image with non-ascii characters in the
name.
This is required for python 2.6 compatibility.
Change-Id: I66ebc27edf4ccd8f903399da58705711c372536d
Closes-Bug: 1448080
We have a basic implementation for a fallback mechanism that will use v1
rather than v2 when downloading schema files from glance-api fails.
However, this is not sound. If the schemas are cached already, we won't
check if v2 is available and fail to fallback.
This patch fixes the aforementioned issue by getting the list of
available versions from the server only when the API versions was not
explicitly specified through the CLI. That is, for all commands that
don't pass `--os-image-api-version 2`, we'll check v2's availability and
we'll fallback to v1 if it isn't available.
This patch also changes how we handle `/versions` calls in the client.
The server has been, incorrectly, replying to requests to `/version`
with a 300 error, which ended up in the client re-raising such
exception. While I think 300 shouldn't raise an exception, I think we
should handle that in a spearate patch. Therefore, this patch just
avoids raising such exception when `/version` is explicitly called.
This fallback behaviour and the check on `/versions` will be removed in
future versions of the client. The later depends on this bug[0] being
fixed.
[0] https://bugs.launchpad.net/glance/+bug/1491350
Closes-bug: #1489381
Change-Id: Ibeba6bc86db2a97b8a2b4bd042248464cd792e5e
Custom SSL handling was introduced because disabling SSL layer compression
provided an approximately five fold performance increase in some
cases. Without SSL layer compression disabled the image transfer would be
CPU bound -- with the CPU performing the DEFLATE algorithm. This would
typically limit image transfers to < 20 MB/s. When --no-ssl-compression
was specified the client would not negotiate any compression algorithm
during the SSL handshake with the server which would remove the CPU
bottleneck and transfers could approach wire speed.
In order to support '--no-ssl-compression' two totally separate code
paths exist depending on whether this is True or False. When SSL
compression is disabled, rather than using the standard 'requests'
library, we enter some custom code based on pyopenssl and httplib in
order to disable compression.
This patch/spec proposes removing the custom code because:
* It is a burden to maintain
Eg adding new code such as keystone session support is more complicated
* It can introduce additional failure modes
We have seen some bugs related to the 'custom' certificate checking
* Newer Operating Systems disable SSL for us.
Eg. While Debian 7 defaulted to compression 'on', Debian 8 has compression
'off'. This makes both servers and client less likely to have compression
enabled.
* Newer combinations of 'requests' and 'python' do this for us
Requests disables compression when backed by a version of python which
supports it (>= 2.7.9). This makes clients more likely to disable
compression out-of-the-box.
* It is (in principle) possible to do this on older versions too
If pyopenssl, ndg-httpsclient and pyasn1 are installed on older
operating system/python combinations, the requests library should
disable SSL compression on the client side.
* Systems that have SSL compression enabled may be vulnerable to the CRIME
(https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4929) attack.
Installations which are security conscious should be running the Glance
server with SSL disabled.
Full Spec: https://review.openstack.org/#/c/187674
Blueprint: remove-custom-client-ssl-handling
Change-Id: I7e7761fc91b0d6da03939374eeedd809534f6edf
This commit enables new flake8 checks:
* E265 block comment should start with '# '
* H405 multi line docstring summary not separated with an empty line
* E123 closing bracket does not match indentation of opening bracket's line
* H238 old style class declaration, use new style (inherit from `object`)
* E128 continuation line under-indented for visual indent
and makes related changes in the code.
Change-Id: Ie993afc930f6b74d7a990bcaa9fc0e9f5ba1585c
As stated in the OpenStack Hacking Guidelines, it is prefered
that only modules should be imported.
Also updated tox.ini to ignore opestack/common among others.
Change-Id: I2f0a603c31052eadee581c11880c0ec6bd392829
To make this work we create a different HTTPClient that extends the
basic keystoneclient Adapter. The Adapter is a standard set of
parameters that all clients should know how to use like region_name and
user_agent. We extend this with the glance specific response
manipulation like loading and sending iterables.
Implements: bp session-objects
Change-Id: Ie8eb4bbf7d1a037099a6d4b272cab70525fbfc85
The oslo.utils libraries are moving away from namespace packages.
This requires oslo.utils>=1.2.0
bp drop-namespace-packages
Change-Id: I803df61e91eabb96329d859aef6bea03530fb84f
Some generated header values may in fact be None. Trying to encode None
causes the client to fail with an exception and cannot be worked around
by the user.
Change-Id: I638b1fba0ef9a07d726445d8c2cdd774140f5b83
Closes-bug: 1415935
This module now lives in oslo.utils, so import it from there instead.
Co-Authored-By: Ian Cordasco <ian.cordasco@rackspace.com>
Change-Id: Ib35dc840992433542490670781badd9529ec8947
If we don't explicitly close a response after streaming its download,
then we can run into HTTPConnectionPool full warnings. It also will hurt
performance if we have to continuously create new sockets for new
responses. Calling close will return the connection to the pool so it
can be reused. Note this is only necessary when streaming a response. If
we don't stream it, then requests will return the connection to the pool
for us.
Change-Id: I803bd4dd0e769c233501d5e5ff07a19705fbe233
Closes-bug: 1341777
The returns from requests' response.content is a bytes type. Under
python 3 this fails in error handling and string conversion. The
response.text variable should be used to treat a response body as a
string.
Closes-Bug: #1407531
Change-Id: Ifd588b5f6820ef21beb186d88d0b3f1a267695aa
When file size is an exact multiple of chunk_size, glance client
is processing EOF in image-data as a chunk and sends to
glance-api. The server treats this as the end of chunked
transmission and sends a http response.
When the actual last chunk is sent by the 'requests' library, the
server sends a 400 response and tracebacks with broken pipe
as the client has already closed the socket.
Closes-Bug: #1342080
Change-Id: Icdbff838450db1c252ddc919a230a7d3ca16765f
Change I09f70eee3e2777f52ce040296015d41649c2586a, introduced a bug where
the identity_headers are not added to the request headers anymore
causing the former to be completely ignored and useless.
This patch fixes that issue by restoring the previous code. A new test
has been added to avoid regressions.
Closes-bug: #1394965
Change-Id: I1b1633636448398cf3f41217f1d671b43ebd9946
imacdonn