python: HTTP PUT with unencoded binary data

Jason S picture Jason S · Jan 2, 2012 · Viewed 11.7k times · Source

I cannot for the life of me figure out how to perform an HTTP PUT request with verbatim binary data in Python 2.7 with the standard Python libraries.

I thought I could do it with urllib2, but that fails because urllib2.Request expects its data in application/x-www-form-urlencoded format. I do not want to encode the binary data, I just want to transmit it verbatim, after the headers that include

Content-Type: application/octet-stream
Content-Length: (whatever my binary data length is)

This seems so simple, but I keep going round in circles and can't seem to figure out how.

How can I do this? (aside from open up a raw binary socket and write to it)

Answer

Jason S picture Jason S · Jan 3, 2012

I found out my problem. It seems there is some obscure behavior in urllib2.Request / urllib2.urlopen() (at least in Python 2.7)

The urllib2.Request(url, data, headers) constructor seems to expect the same type of string in its url and data parameters.

I was giving the data parameter raw data from a file read() call (which in Python 2.7 returns it in the form of a 'plain' string), but my url was accidentally Unicode because I concatenated a portion of the URL from the result of another function which returned Unicode strings.

Rather than trying to "downcast" url from Unicode -> plain strings, it tried to "upcast" the data parameter to Unicode, and it gave me a codec error. (oddly enough, this happens on the urllib2.urlopen() function call, not the urllib2.Request constructor)

When I changed my function call to

# headers contains `{'Content-Type': 'application/octet-stream'}`
r = urllib2.Request(url.encode('utf-8'), data, headers)

it worked fine.