Scrape web pages in real time with Node.js

javascript jquery node.js screen-scraping web-scraping
Avishai picture Avishai · Mar 6, 2011 · Viewed 44.2k times · Source

What's a good was to scrape website content using Node.js. I'd like to build something very, very fast that can execute searches in the style of kayak.com, where one query is dispatched to several different sites, the results scraped, and returned to the client as they become available.

Let's assume that this script should just provide the results in JSON format, and we can process them either directly in the browser or in another web application.

A few starting points:

Using node.js and jquery to scrape websites

Anybody have any ideas?

Answer

Avishai picture Avishai · Mar 12, 2011

Node.io seems to take the cake :-)

Related questions

Can I use jQuery with Node.js?

Is it possible to use jQuery selectors/DOM manipulation on the server-side using Node.js?

Read More
DataTables: Cannot read property style of undefined

I am getting this error with the following: jquery.dataTables.js:4089 Uncaught TypeError: Cannot read property 'style' of undefined(…) _fnCalculateColumnWidths @ jquery.dataTables.js:4089 _fnInitialise @ jquery.dataTables.js:3216 (anonymous function) @ jquery.dataTables.js:6457 each @ jquery-2.0.2.min.js:4 each @ jquery-2.0.2.min.js:4 …

Read More
export html table to csv

I am trying to add a feature of csv download option in my website. It should convert the html table present in the website in to csv content and make it downloadable. Ive been searching through internet for a good …

Read More