How to combine scrapy and htmlunit to crawl urls with javascript

javascript htmlunit scrapy
HjySix picture HjySix · Nov 8, 2011 · Viewed 8.9k times · Source

I'm working on Scrapy to crawl pages,however,I can't handle the pages with javascript. People suggest me to use htmlunit, so I got it installed,but I don't know how to use it at all.Dose anyone can give an example(scrapy + htmlunit) for me? Thanks very much.

Answer

reclosedev picture reclosedev · Nov 17, 2011

To handle the pages with javascript you can use Webkit or Selenium.

Here some snippets from snippets.scrapy.org:

Rendered/interactive javascript with gtk/webkit/jswebkit

Rendered Javascript Crawler With Scrapy and Selenium RC

Related questions

How to overcome an HTMLUnit ScriptException?

I got a problem with a line of code that probably triggers some js function e couse an exception, how can i fix this? box.setText(link.toString()); client.waitForBackgroundJavaScriptStartingBefore(10000); box.dblClick(); //this line cause the exception Exception in thread "…

Read More
calling a JavaScript function with HTMLUnit

I'm trying to call the function showPage('3'); of this page, for use the page source code after. I tried to do with htmlUnit like so: WebClient webClient = new WebClient(); webClient.waitForBackgroundJavaScriptStartingBefore(10000); HtmlPage page = webClient.getPage("http://www.visittrentino.…

Read More
Accessing html generated by Javascript with htmlunit -Java

I am trying to be able to test a website that uses javascript to render most of the HTML. With the HTMLUNIT browser how would you be able to access the html generated by the javascript? I was looking through …

Read More