Web crawler that can interpret JavaScript

javascript web-crawler
user320662 picture user320662 · Apr 19, 2010 · Viewed 20.4k times · Source

I want to write a web crawler that can interpret JavaScript. Basically its a program in Java or PHP that takes a URL as input and outputs the DOM tree which is similar to the output in Firebug HTML window. The best example is Kayak.com where you can not see the resulting DOM displayed on the browser when you 'view source' but can save the resulting HTML though Firebug.

How would I go about doing this? What tools exist that would help me?

Answer

tokland picture tokland · Oct 17, 2011

Ruby's Capybara is an integration test library, but it can also be used to write stand-alone web-crawlers. Given that it uses backends like Selenium or headless WebKit, it interprets javascript out-of-the-box:

require 'capybara/dsl'
require 'capybara-webkit'

include Capybara::DSL
Capybara.current_driver = :webkit
Capybara.app_host = "http://www.google.com"
page.visit("/")
puts(page.html)

Related questions

Node.JS: How to pass variables to asynchronous callbacks?

I'm sure my problem is based on a lack of understanding of asynch programming in node.js but here goes. For example: I have a list of links I want to crawl. When each asynch request returns I want to …

Read More
Detect Search Crawlers via JavaScript

I am wondering how would I go abouts in detecting search crawlers? The reason I ask is because I want to suppress certain JavaScript calls if the user agent is a bot. I have found an example of how to …

Read More
How can I scrape pages with dynamic content using node.js?

I am trying to scrape a website but I don't get some of the elements, because these elements are dynamically created. I use the cheerio in node.js and My code is below. var request = require('request'); var cheerio = require(…

Read More