Robots.txt: allow only major SE

web-crawler robots.txt
vyger picture vyger · Mar 22, 2009 · Viewed 22.7k times · Source

Is there a way to configure the robots.txt so that the site accepts visits ONLY from Google, Yahoo! and MSN spiders?

Answer

NoahD picture NoahD · Mar 22, 2009 

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Slurp
Allow: /
User-Agent: msnbot
Disallow:

Slurp is Yahoo's robot

Related questions

Robots.txt - What is the proper format for a Crawl Delay for multiple user agents?

Below is a sample robots.txt file to Allow multiple user agents with multiple crawl delays for each user agent. The Crawl-delay values are for illustration purposes and will be different in a real robots.txt file. I have searched …

Read More
Ban robots from website

my website is often down because a spider is accessying to many resources. This is what the hosting told me. They told me to ban these IP address: 46.229.164.98 46.229.164.100 46.229.164.101 But I've no idea about how to do this. I've googled a …

Read More
Block a site from search engine - DuckDuckGo

I have a development site https://text-domain.com. (not a real site) When I go to https://duckduckgo.com and search for text-domain.com, it does return results. What have I tried so far: Created robots.txt file with following …

Read More