I am trying to send a POST using mechanize however my code won't work sometimes(and I know why). I have used mechanize, twill and requests. With mechanize and twills it is working and with requests it is not. Probably I am doing it in wrong way.
My mechanize code. The following works :
#!/usr/bin/env python
import sys
import urllib
import mechanize
from mechanize import ParseResponse, urlopen, urljoin
response = urlopen(url)
forms = ParseResponse(response, backwards_compat=False)
form = forms[0]
form["username"] = "avi"
form["password"] = "stackoverflow"
urlopen(form.click())
My twill code. It also works :
import os
import twill
from twill.commands import *
out = open(os.devnull,"w")
twill.set_output(out)
go(url)
formvalue("1", "username", "avi")
formvalue("1", "password", "stackoverflow")
submit()
go(url2) #some protected page
content = show()
print 'content is',content[:100]
From what I understand, mechanize or twill first gets the page, fills the form and sends the form. But problem here is, sometimes target page displays random page, without any form. In that case I get an error, obviously because there is no form to show. I don't want to handle this error, because I already know the post URL. even though some random page is displayed, after I click next, same form is loaded. The POST url, fields all remain same. So I wanted to send the POST request directly, since I already know details everything required. Here is my code, based on :
browser = mechanize.Browser()
parameters = {"username" : "avi",
"password" : "stackoverflow",
}
data = urllib.urlencode(parameters)
browser.open(post_url,data)
cool = browser.open(post_url + '%s' % data).read()
print cool
I get an error :
urllib2.URLError: <urlopen error [Errno 61] Connection refused>
I tried same thing with requests. But I am getting same error as above. Here is the code :
import requests
from requests import session
payload = { 'username': 'avi','password': 'stackoverflow'}
url1 = 'http://example.com/login.php'
url2 = 'http://example.com/protected.php'
with session() as c:
c.post(url1, data=payload)
c.get(url2)
So I poked here and there, read few more code available online and I think it is not working because to prevent CSRF. So I went to same page with twill, did showforms and saw there is some value with token :
Here is what I want :
I've used the code below with success:
params = {u'user_login': self.USER, u'password':self.PASSWORD}
data = urllib.urlencode(params)
request = mechanize.Request( loginURL )
response = mechanize.urlopen(request, data=data)
To handle the CSRF token issue, I added the line below to my controller:
skip_before_filter :verify_authenticity_token, :only => [:create]
To keep the session between requests, I use a cookiesJar. But you can retrieve the CSRF token from your request and keep it in your application, adding it the each new request.