Headless Browser for Python (Javascript support REQUIRED!)

javascript python screen-scraping headless-browser
Steven Matthews picture Steven Matthews · May 17, 2011 · Viewed 39.7k times · Source

I need a headless browser which is fairly easy to use (I am still fairly new to Python and programming in general) which will allow me to navigate to a page, log into a form that requires Javascript, and then scrape the resulting web page by searching for results matching certain criteria, clicking check boxes, and clicking to download files. All of this requires Javascript.

I hear a headless browser is what I want - requirements/preferences are that I be able to run it from Python, and preferably that the resultant script will be compilable by py2exe (I am writing this program for other users).

So far Windmill looks like it MIGHT be what I want, but I am not sure.

Any ideas appreciated!

Answer

hoju picture hoju · May 17, 2011

I use webkit as a headless browser in Python via pyqt / pyside:
http://www.riverbankcomputing.co.uk/software/pyqt/download
http://developer.qt.nokia.com/wiki/Category:LanguageBindings::PySide::Downloads

I particularly like webkit because it is simple to setup. For Ubuntu you just use: sudo apt-get install python-qt4

Here is an example script:
http://webscraping.com/blog/Scraping-JavaScript-webpages-with-webkit/

Related questions

Can scrapy be used to scrape dynamic content from websites that are using AJAX?

I have recently been learning Python and am dipping my hand into building a web-scraper. It's nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel. …

Read More
Executing Javascript from Python

I have HTML webpages that I am crawling using xpath. The etree.tostring of a certain node gives me this string: <script> <!-- function escramble_758(){ var a,b,c a='+1 ' b='84-' a+='425…

Read More
Scraping websites with Javascript enabled?

I'm trying to scrape and submit information to websites that heavily rely on Javascript to do most of its actions. The website won't even work when i disable Javascript in my browser. I've searched for some solutions on Google and …

Read More