Is there a list of known web crawlers?

list documentation web-crawler bots
pupeno picture pupeno · Nov 14, 2009 · Viewed 14.8k times · Source

I'm trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I'm not sure, they may or may not be a web crawler and they are causing many downloads so it's important for me to know.

Is there somewhere a list of know web crawlers with some documentation like user agent, IPs, behavior, etc?

I'm not interested in the official ones, like Google's, Yahoo's, or Microsoft's. Those are generally well behaved and self-indentified.

Answer

Jaan J picture Jaan J · Nov 14, 2009

I'm using http://www.user-agents.org/ usually as reference, hope this helps you out.

You can also try http://www.robotstxt.org/db.html or http://www.botsvsbrowsers.com.

Related questions

Finding the index of an item in a list

Given a list ["foo", "bar", "baz"] and an item in the list "bar", how do I get its index (1) in Python?

Read More
How do I get the number of elements in a list?

Consider the following: items = [] items.append("apple") items.append("orange") items.append("banana") # FAKE METHOD: items.amount() # Should return 3 How do I get the number of elements in the list items?

Read More
How do I check if a list is empty?

For example, if passed the following: a = [] How do I check to see if a is empty?

Read More