Ruby on Rails, How to determine if a request was made by a robot or search engine spider?

ruby-on-rails ruby-on-rails-3 search-engine web-crawler
Agung Prasetyo picture Agung Prasetyo · May 4, 2011 · Viewed 8.2k times · Source

I've Rails apps, that record an IP-address from every request to specific URL, but in my IP database i've found facebook blok IP like 66.220.15.* and Google IP (i suggest it come from bot). Is there any formula to determine an IP from request was made by a robot or search engine spider ? Thanks

Answer

tribalvibes picture tribalvibes · Feb 15, 2012

Since the well behaved bots at least typically include a reference URI in the UA string they send, something like:

request.env["HTTP_USER_AGENT"].match(/\(.*https?:\/\/.*\)/)

is an easy way to see if the request is from a bot vs. a human user's agent. This seems to be more robust than trying to match against a comprehensive list.

Related questions

How to get a random number in Ruby

How do I generate a random number between 0 and n?

Read More
Rails: How can I rename a database column in a Ruby on Rails migration?

I wrongly named a column hased_password instead of hashed_password. How do I update the database schema, using migration to rename this column?

Read More
How to remove a key from Hash and get the remaining hash in Ruby/Rails?

To add a new pair to Hash I do: {:a => 1, :b => 2}.merge!({:c => 3}) #=> {:a => 1, :b => 2, :c => 3} Is there a similar way to delete a key from Hash ? This works: {:a => 1, :b => 2}.…

Read More