robots.txt to disallow all pages except one? Do they override and cascade?

nouveau picture nouveau · Nov 8, 2013 · Viewed 37.6k times · Source

I want one page of my site to be crawled and no others.

Also, if it's any different than the answer above, I would also like to know the syntax for disallowing everything but the root (index) of the website is.

# robots.txt for http://example.com/

User-agent: *
Disallow: /style-guide
Disallow: /splash
Disallow: /etc
Disallow: /etc
Disallow: /etc
Disallow: /etc
Disallow: /etc

Or can I do like this?

# robots.txt for http://example.com/

User-agent: *
Disallow: /
Allow: /under-construction

Also I should mention that this is a WordPress install, so "under-construction," for example, is set to the front page. So in that case it acts as the index.

I think what I need is to have http://example.com craweld, but no other pages.

# robots.txt for http://example.com/

User-agent: *
Disallow: /*

Would this mean disallow anything after the root?

Answer

Jim Mischel picture Jim Mischel · Nov 8, 2013

The easiest way to allow access to just one page would be:

User-agent: *
Allow: /under-construction
Disallow: /

The original robots.txt specification says that crawlers should read robots.txt from top to bottom, and use the first matching rule. If you put the Disallow first, then many bots will see it as saying they can't crawl anything. By putting the Allow first, those that apply the rules from top to bottom will see that they can access that page.

The expression rules are simple: the expression Disallow: / says "disallow anything that starts with a slash." So that means everything on the site.

Your Disallow: /* means the same thing to Googlebot and Bingbot, but bots that don't support wildcards could see the /* and think that you meant a literal *. So they could assume that it was okay to crawl /*foo/bar.html.

If you just want to crawl http://example.com, but nothing else, you might try:

Allow: /$
Disallow: /

The $ means "end of string," just like in regular expressions. Again, that'll work for Google and Bing, but won't work for other crawlers if they don't support wildcards.