How do screen scrapers work?

screen-scraping web-scraping html-content-extraction pdf-scraping console-scraping
Micah picture Micah · Oct 1, 2008 · Viewed 22k times · Source

I hear people writing these programs all the time and I know what they do, but how do they actually do it? I'm looking for general concepts.

Answer

bmdhacks picture bmdhacks · Oct 1, 2008

Technically, screenscraping is any program that grabs the display data of another program and ingests it for it's own use.

Quite often, screenscaping refers to a web client that parses the HTML pages of targeted website to extract formatted data. This is done when a website does not offer an RSS feed or a REST API for accessing the data in a programmatic way.

One example of a library used for this purpose is Hpricot for Ruby, which is one of the better-architected HTML parsers used for screen scraping.

Related questions

Web scraping with Python

I'd like to grab daily sunrise/sunset times from a web site. Is it possible to scrape web content with Python? what are the modules used? Is there any tutorial available?

Read More
What's the best way of scraping data from a website?

I need to extract contents from a website, but the application doesn’t provide any application programming interface or another mechanism to access that data programmatically. I found a useful third-party tool called Import.io that provides click and go …

Read More
Scrape web page contents

I am developing a project, for which I want to scrape the contents of a website in the background and get some limited content from that scraped website. For example, in my page I have "userid" and "password" fields, by …

Read More