Click on a javascript link within python?

javascript python screen-scraping mechanize spidermonkey
Lostsoul picture Lostsoul · Mar 6, 2011 · Viewed 11.6k times · Source

I am navigating a site using python's mechanize module and having trouble clicking on a javascript link for next page. I did a bit of reading and people suggested I need python-spidermonkey and DOMforms. I managed to get them installed by I am not sure of the syntax to actually click on the link.

I can identify the code on the page as: 

<a href="javascript:__doPostBack('ctl00$MainContent$gvSearchResults','Page$2')">2</a>

Does anyone know how to click on it? or if perhaps there's another tool.

Thanks

Answer

sw. picture sw. · Mar 8, 2011

I mainly use HtmlUnit under jython for these use cases. Also I published a simple article on the subject: Web Scraping Ajax and Javascript sites.

Related questions

Programmatic Python Browser with JavaScript

I want to screen-scrape a web-site that uses JavaScript. There is mechanize, the programmatic web browser for Python. However, it (understandably) doesn't interpret javascript. Is there any programmatic browser for Python which does? If not, is there any JavaScript implementation …

Read More
Can scrapy be used to scrape dynamic content from websites that are using AJAX?

I have recently been learning Python and am dipping my hand into building a web-scraper. It's nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel. …

Read More
Executing Javascript from Python

I have HTML webpages that I am crawling using xpath. The etree.tostring of a certain node gives me this string: <script> <!-- function escramble_758(){ var a,b,c a='+1 ' b='84-' a+='425…

Read More