Anyone know of a good Python based web crawler that I could use?

python web-crawler
Matt picture Matt · Jan 7, 2009 · Viewed 95.8k times · Source

I'm half-tempted to write my own, but I don't really have enough time right now. I've seen the Wikipedia list of open source crawlers but I'd prefer something written in Python. I realize that I could probably just use one of the tools on the Wikipedia page and wrap it in Python. I might end up doing that - if anyone has any advice about any of those tools, I'm open to hearing about them. I've used Heritrix via its web interface and I found it to be quite cumbersome. I definitely won't be using a browser API for my upcoming project.

Thanks in advance. Also, this is my first SO question!

Answer

RexE picture RexE · Jan 7, 2009
  • Mechanize is my favorite; great high-level browsing capabilities (super-simple form filling and submission).
  • Twill is a simple scripting language built on top of Mechanize
  • BeautifulSoup + urllib2 also works quite nicely.
  • Scrapy looks like an extremely promising project; it's new.

Related questions

Sending "User-agent" using Requests library in Python

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below: debug = {'verbose': …

Read More
TypeError: can't use a string pattern on a bytes-like object in re.findall()

I am trying to learn how to automatically fetch urls from a page. In the following code I am trying to get the title of the webpage: import urllib.request import re url = "http://www.google.com" regex = r'<…

Read More
Python: maximum recursion depth exceeded while calling a Python object

I've built a crawler that had to run on about 5M pages (by increasing the url ID) and then parses the pages which contain the info' I need. after using an algorithm which run on the urls (200K) and saved …

Read More