urlparse.parse_qs
is usefull for parsing url parameters, and it works fine with simple ASCII url, represented by str
. So i can parse a query and then construct the same path using urllib.urlencode
from parsed data:
>>> import urlparse
>>> import urllib
>>>
>>> path = '/?key=value' #path is str
>>> query = urlparse.urlparse(path).query
>>> query
'key=value'
>>> query_dict = urlparse.parse_qs(query)
>>> query_dict
{'key': ['value']}
>>> '/?' + urllib.urlencode(query_dict, doseq=True)
'/?key=value' # <-- path is the same here
It also works fine, when url contains percent encoded non-ASCII param:
>>> value = urllib.quote(u'значение'.encode('utf8'))
>>> value
'%D0%B7%D0%BD%D0%B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5'
>>> path = '/?key=%s' % value
>>> path
'/?key=%D0%B7%D0%BD%D0%B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5'
>>> query = urlparse.urlparse(path).query
>>> query
'key=%D0%B7%D0%BD%D0%B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5'
>>> query_dict = urlparse.parse_qs(query)
>>> query_dict
{'key': ['\xd0\xb7\xd0\xbd\xd0\xb0\xd1\x87\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb5']}
>>> '/?' + urllib.urlencode(query_dict, doseq=True)
'/?key=%D0%B7%D0%BD%D0%B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5' # <-- path is the same here
But, when using django, i get the url using request.get_full_path()
, and it returns path as unicode
string:
>>> path = request.get_full_path()
>>> path
u'/?key=%D0%B7%D0%BD%D0%B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5' # path is unicode
Look what will happen now:
>>> query = urlparse.urlparse(path).query
>>> query
u'key=%D0%B7%D0%BD%D0%B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5'
>>> query_dict = urlparse.parse_qs(query)
>>> query_dict
{u'key': [u'\xd0\xb7\xd0\xbd\xd0\xb0\xd1\x87\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb5']}
>>>
query_dict
contains unicode string, that contains bytes! Not unicode points!
And of course i've got a UnicodeEncodeError, when trying to urlencode that string:
>>> urllib.urlencode(query_dict, doseq=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\Lib\urllib.py", line 1337, in urlencode
l.append(k + '=' + quote_plus(str(elt)))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)
Currently i have a solution:
# just convert path, returned by request.get_full_path(), to `str` explicitly:
path = str(request.get_full_path())
So the questions are:
Encode back to bytes before passing it to .parse_qs()
, using ASCII:
query_dict = urlparse.parse_qs(query.encode('ASCII'))
This does the same thing as str()
but with an explicit encoding. Yes, this is safe, the URL encoding uses ASCII codepoints only.
parse_qs
was handed a Unicode value, so it returned you a unicode value too; it is not it's job to decode bytes.