how to restrict the site from being indexed

.htaccess search indexing robots.txt
Ilian Andreev picture Ilian Andreev · May 26, 2012 · Viewed 12.6k times · Source

I know this question was being asked many times but I want to be more specific.

I have a development domain and moved the site there to a subfolder. Let's say from:

http://www.example.com/

To:

http://www.example.com/backup

So I want the subfolder to not be indexed by search engines at all. I've put robots.txt with the following contents in the subfolder (can I put it in a subfolder or it has to be at the root always, because I want the content at the root to be visible to search engines):

User-agent: *
Disallow: /

Maybe I need to replace it and put in the root the following:

User-agent: *
Disallow: /backup

The other thing is, I read somewhere that certain robots don't respect the robots.txt file so would just putting an .htaccess file in the /backup folder do the job?

Order deny,allow
Deny from all

Any ideas?

Answer

jmort253 picture jmort253 · May 26, 2012

This would prevent that directory from being indexed:

User-agent: *
Disallow: /backup/

Additionally, your robots.txt file must be placed in the root of your domain, so in this case, the file would be placed where you can access it in your browser by going to http://example.com/robots.txt

As an aside, you may want to consider setting up a subdomain for your development site, something like http://dev.example.com. Doing so would allow you to completely separate the dev stuff from the production environment and would also ensure that your environments more closely match.

For instance, any absolute paths to JavaScript files, CSS, images or other resources may not work the same from dev to production, and this may cause some issues down the road.

For more information on how to configure this file, see the robotstxt.org site. Good luck!

As a last and final note Google Webmaster Tools has a section where you can see what is blocked by the robots.txt file:

To see which URLs Google has been blocked from crawling, visit the Blocked URLs page of the Health section of Webmaster Tools.

I strongly suggest you use this tool, as an incorrectly configured robots.txt file could have a significant impact on the performance of your website.

Related questions

Access-Control-Allow-Origin Multiple Origin Domains?

Is there a way to allow multiple cross-domains using the Access-Control-Allow-Origin header? I'm aware of the *, but it is too open. I really want to allow just a couple domains. As an example, something like this: Access-Control-Allow-Origin: http://domain1.example, …

Read More
How to redirect all HTTP requests to HTTPS

I'm trying to redirect all insecure HTTP requests on my site (e.g. http://www.example.com) to HTTPS (https://www.example.com). I'm using PHP btw. Can I do this in .htaccess?

Read More
htaccess redirect to https://www

I have the following htaccess code: <IfModule mod_rewrite.c> RewriteEngine On RewriteCond !{HTTPS} off RewriteRule ^(.*)$ https://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301] RewriteCond %{HTTP_HOST} !^www\. RewriteRule ^(.*)$ https://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301] </…

Read More