scrape html generated by javascript with python

javascript python browser screen-scraping
hymloth picture hymloth · Jan 27, 2010 · Viewed 15.5k times · Source

I need to scrape a site with python. I obtain the source html code with the urlib module, but I need to scrape also some html code that is generated by a javascript function (which is included in the html source). What this functions does "in" the site is that when you press a button it outputs some html code. How can I "press" this button with python code? Can scrapy help me? I captured the POST request with firebug but when I try to pass it on the url I get a 403 error. Any suggestions?

Answer

Paul D. Waite picture Paul D. Waite · Mar 11, 2011

In Python, I think Selenium 1.0 is the way to go. It’s a library that allows you to control a real web browser from your language of choice.

You need to have the web browser in question installed on the machine your script runs on, but it looks like the most reliable way to programmatically interrogate websites that use a lot of JavaScript.

Related questions

Programmatic Python Browser with JavaScript

I want to screen-scrape a web-site that uses JavaScript. There is mechanize, the programmatic web browser for Python. However, it (understandably) doesn't interpret javascript. Is there any programmatic browser for Python which does? If not, is there any JavaScript implementation …

Read More
Mechanize and Javascript

I want to use Mechanize to simulate browsing to a web page with active JavaScript, including DOM Events and AJAX, and so far I've found no way to do that. I looked at some Python client browsers that support JavaScript …

Read More
"The owner of this website has banned your access based on your browser's signature" ... on a url request in a python program

When doing a simple request, on python (Entought Canopy to be precise), with urllib2, the server denies me access : data = urllib.urlopen(an url i cannot post because of reputation, params) print data.read() Error: Access denied | play.pokemonshowdown.com …

Read More