How to configure robots.txt to allow everything?

robots.txt
Raajpoot picture Raajpoot · Nov 25, 2010 · Viewed 125.7k times · Source

My robots.txt in Google Webmaster Tools shows the following values:

User-agent: *
Allow: /

What does it mean? I don't have enough knowledge about it, so looking for your help. I want to allow all robots to crawl my website, is this the right configuration?

Answer

Jim picture Jim · Nov 25, 2010

That file will allow all crawlers access

User-agent: *
Allow: /

This basically allows all user agents (the *) to all parts of the site (the /).

Related questions

Can a relative sitemap url be used in a robots.txt?

In robots.txt can I write the following relative URL for the sitemap file? sitemap: /sitemap.ashx Or do I have to use the complete (absolute) URL for the sitemap file, like: sitemap: http://subdomain.domain.com/sitemap.ashx Why …

Read More
Ignore urls in robot.txt with specific parameters?

I would like for google to ignore urls like this: http://www.mydomain.com/new-printers?dir=asc&order=price&p=3 All urls that have the parameters dir, order and price should be ignored but I dont have experience …

Read More
How do I disallow specific page from robots.txt

I am creating two pages on my site that are very similar but serve different purposes. One is to thank users for leaving a comment and the other is to encourage users to subscribe. I don't want the duplicate content …

Read More