how to ban crawler 360Spider with robots.txt or .htaccess?

.htaccess search-engine web-crawler bots robots.txt
kovpack picture kovpack · Sep 6, 2012 · Viewed 13.1k times · Source

I've got a problems because of 360Spider: this bot makes too many requests per second to my VPS and slows it down (the CPU-usage becomes 10-70%, but usually i have 1-2%). I looked into httpd logs and saw there such lines:

182.118.25.209 - - [06/Sep/2012:19:39:08 +0300] "GET /slovar/znachenie-slova/42957-polovity.html HTTP/1.1" 200 96809 "http://www.hrinchenko.com/slovar/znachenie-slova/42957-polovity.html" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
182.118.25.208 - - [06/Sep/2012:19:39:08 +0300] "GET /slovar/znachenie-slova/52614-rospryskaty.html HTTP/1.1" 200 100239 "http://www.hrinchenko.com/slovar/znachenie-slova/52614-rospryskaty.html" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider

etc.

How can I block this spider completely via robots.txt? Now my robots.txt looks like this:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/

User-agent: YoudaoBot
Disallow: /

User-agent: sogou spider
Disallow: /

I've added lines:

User-agent: 360Spider
Disallow: /

but that does not seem to work. How to block this angry bot?

If you offer to block it via .htaccess, so mind that it looks now like this:

# Turn on URL rewriting
RewriteEngine On

# Installation directory
RewriteBase /

SetEnvIfNoCase Referer ^360Spider$ block_them
Deny from env=block_them

# Protect hidden files from being viewed
<Files .*>
    Order Deny,Allow
    Deny From All
</Files>

# Protect application and system files from being viewed
RewriteRule ^(?:application|modules|system)\b.* index.php/$0 [L]

# Allow any files or directories that exist to be displayed directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

# Rewrite all other URLs to index.php/URL
RewriteRule .* index.php/$0 [PT]

And, in spite of presence of 

SetEnvIfNoCase Referer ^360Spider$ block_them
Deny from env=block_them

this bot still tries to kill my VPS and is logged in access logs.

Answer

Sloth picture Sloth · Jan 26, 2013

In your .htaccess file simply add the following :

RewriteCond %{REMOTE_ADDR} ^(182\.118\.2)

RewriteRule ^.*$ http://182.118.25.209/take_a_hike_moron [R=301,L]

This will catch ALL the bots being launched from the 182.118.2xx.xxx range and send them back to themself...

The crappy 360 bot is being fired from servers in China... so as long as you don't mind saying bye bye to crappy Chinese traffic from that IP range, this will guaranteed make those puppies disappear from reaching any files on your web site.

The following two lines in your .htaccess file will also pick it off simply by it being stupid enough to proudly put 360spider in its user agent string. This could be handy for when they use other IP ranges then the 182.118.2xx.xxx

RewriteCond %{HTTP_USER_AGENT} .*(360Spider) [NC]

RewriteRule ^.*$ http://182.118.25.209/take_a_hike_moron [R=301,L]

And yes... I hate them too !

Related questions

How do I tell search engines not to index content via secondary domain names?

I have a website at a.com (for example). I also have a couple of other domain names which I am not using for anything: b.com and c.com. They currently forward to a.com. I have noticed that …

Read More
Access-Control-Allow-Origin Multiple Origin Domains?

Is there a way to allow multiple cross-domains using the Access-Control-Allow-Origin header? I'm aware of the *, but it is too open. I really want to allow just a couple domains. As an example, something like this: Access-Control-Allow-Origin: http://domain1.example, …

Read More
How to redirect all HTTP requests to HTTPS

I'm trying to redirect all insecure HTTP requests on my site (e.g. http://www.example.com) to HTTPS (https://www.example.com). I'm using PHP btw. Can I do this in .htaccess?

Read More