What is the difference between web-crawling and web-scraping?

search-engine web-scraping web-crawler
wassimans picture wassimans · Dec 1, 2010 · Viewed 65.5k times · Source

Is there a difference between Crawling and Web-scraping?

If there's a difference, what's the best method to use in order to collect some web data to supply a database for later use in a customised search engine?

Answer

Ben picture Ben · Dec 1, 2010

Crawling would be essentially what Google, Yahoo, MSN, etc. do, looking for ANY information. Scraping is generally targeted at certain websites, for specfic data, e.g. for price comparison, so are coded quite differently.

Usually a scraper will be bespoke to the websites it is supposed to be scraping, and would be doing things a (good) crawler wouldn't do, i.e.:

  • Have no regard for robots.txt
  • Identify itself as a browser
  • Submit forms with data
  • Execute Javascript (if required to act like a user)

Related questions

How to find Google's IP address?

Google is blocked in some countries. However, there are many ways to access Google, like VPN, agent, and by changing the hosts file. If I want to change the hosts file to access Google, how can I find an available …

Read More
How do search engines deal with AngularJS applications?

I see two issues with AngularJS application regarding search engines and SEO: 1) What happens with custom tags? Do search engines ignore the whole content within those tags? i.e. suppose I have <custom> <h1>Hey, this …

Read More
Search in html source with GOOGLE?

I have several websites, and I can't remember where I wrote some lines of code. As my pages are indexed by Google, I would like to know if Google offers a facility to search within the HTML source code/mark-up …

Read More