Programmatic Python Browser with JavaScript

javascript python browser screen-scraping mechanize
Claudiu picture Claudiu · Dec 16, 2009 · Viewed 17.1k times · Source

I want to screen-scrape a web-site that uses JavaScript.

There is mechanize, the programmatic web browser for Python. However, it (understandably) doesn't interpret javascript. Is there any programmatic browser for Python which does? If not, is there any JavaScript implementation in Python that I could use to attempt to create one?

Answer

Annie picture Annie · Dec 16, 2009

You might be better off using a tool like Selenium to automate the scraping using a web browser, so the JS executes and the page renders just like it would for a real user.

Related questions

scrape html generated by javascript with python

I need to scrape a site with python. I obtain the source html code with the urlib module, but I need to scrape also some html code that is generated by a javascript function (which is included in the html …

Read More
Mechanize and Javascript

I want to use Mechanize to simulate browsing to a web page with active JavaScript, including DOM Events and AJAX, and so far I've found no way to do that. I looked at some Python client browsers that support JavaScript …

Read More
"The owner of this website has banned your access based on your browser's signature" ... on a url request in a python program

When doing a simple request, on python (Entought Canopy to be precise), with urllib2, the server denies me access : data = urllib.urlopen(an url i cannot post because of reputation, params) print data.read() Error: Access denied | play.pokemonshowdown.com …

Read More