Submitting to a web form using python

Serial picture Serial · Jul 7, 2013 · Viewed 97k times · Source

I have seen questions like this asked many many times but none are helpful

Im trying to submit data to a form on the web ive tried requests, and urllib and none have worked

for example here is code that should search for the [python] tag on SO:

import urllib
import urllib2

url = 'http://stackoverflow.com/'

# Prepare the data
values = {'q' : '[python]'}
data = urllib.urlencode(values)

# Send HTTP POST request
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)

html = response.read()

# Print the result
print html

yet when i run it i get the html soure of the home page

here is an example of using requests:

import requests

data= {
    'q': '[python]'
    }
r = requests.get('http://stackoverflow.com', data=data)

print r.text

same result! i dont understand why these methods arent working i've tried them on various sites with no success so if anyone has successfully done this please show me how!

Thanks so much!

Answer

Johnsyweb picture Johnsyweb · Jul 7, 2013

If you want to pass q as a parameter in the URL using requests, use the params argument, not data (see Passing Parameters In URLs):

r = requests.get('http://stackoverflow.com', params=data)

This will request https://stackoverflow.com/?q=%5Bpython%5D , which isn't what you are looking for.

You really want to POST to a form. Try this:

r = requests.post('https://stackoverflow.com/search', data=data)

This is essentially the same as GET-ting https://stackoverflow.com/questions/tagged/python , but I think you'll get the idea from this.