Posting raw data with Python

python post curl urllib2 pycurl
Oli picture Oli · Dec 15, 2011 · Viewed 15.2k times · Source

I'm playing around with the Google Checkout API and I want to pull it into a Django app. I need to post data to Google using basic http authentication. I've been testing this with curl like this:

curl -d "$(cat mytest.xml)" -u username:password https://url

And that posts the content of my test XML file to Google. And it works fine!

But I'm having problems porting that simple line to Python. I've managed several different ways (httplib2, urllib2, pycurl) of connecting with a password and posting something but the respose is always 400 BAD REQUEST.

Is there a python equivalent for posting block of text to a HTTP Basic auth server? I'm running out of walls to bang my head against.

Apologies for not adding any code. Here are some of my greatest hits. In each, DATA is an XML string. URL, USERNAME and PASSWORD are constant.

req = urllib2.Request(URL)
req.add_header("Authorization", "Basic %s" % base64.encodestring('%s:%s'%(USERNAME, PASSWORD)))
u = urllib2.urlopen(req, DATA)

Gives me a lovely HTTP Error 400: Bad Request

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, URL, USERNAME, PASSWORD)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
pagehandle = urllib2.urlopen(URL, DATA)

Gives HTTP Error 401: Unauthorized

pycurl.global_init(pycurl.GLOBAL_DEFAULT)
c = pycurl.Curl()
c.setopt(pycurl.URL, URL)
c.setopt(pycurl.USERPWD, "%s:%s" % (USERNAME,PASSWORD))
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.HTTPHEADER, ["Content-type: text/xml"])
c.setopt(pycurl.POSTFIELDS, DATA)
b = StringIO.StringIO()
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.perform()

Seems to struggle with passing the DATA string as a POSTFIELD. I've tried urllib.urlencode()ing DATA in several different ways but 

h = httplib2.Http()
h.add_credentials(USERNAME, PASSWORD)
print = h.request(URL, "POST", body=base64.encodestring(DATA))

The credentials don't seem to do anything - I get an unauthorised message back from Google.

There are more but they're all based on these.

Answer

nemesisfixx picture nemesisfixx · May 14, 2012

I've been having similar turmoils with the stdlib packages, until somoneone pointed to the awesome requests that supports basic Http Authentication and other authentication means straight out-of-the-box! And it has a beautiful and simple API it hurts!

requests.post(url, data=DATA, headers=HEADERS_DICT, auth=(username, password))

It supports a host of other necessary features (e.g HTTPS, Digest Authentication, etc) Please check it out if u must...

Related questions

Curl POST request into pycurl code

I'm trying to convert following curl request into pycurl: curl -v -H Accept:application/json \ -H Content-Type:application/json \ -d "{ name: 'abc', path: 'def', target: [ 'ghi' ] }" \ -X POST http://some-url I have following python code: import pycurl, json c = pycurl.…

Read More
How to get POSTed JSON in Flask?

I'm trying to build a simple API using Flask, in which I now want to read some POSTed JSON. I do the POST with the Postman Chrome extension, and the JSON I POST is simply {"text":"lalala"}. I try to …

Read More
How do I send a POST request as a JSON?

data = { 'ids': [12, 3, 4, 5, 6 , ...] } urllib2.urlopen("http://abc.com/api/posts/create",urllib.urlencode(data)) I want to send a POST request, but one of the fields should be a list of numbers. How can I do that ? (JSON?)

Read More