Can I prevent search engines from indexing an entire directory on my website?

staging
Marty picture Marty · Jan 29, 2012 · Viewed 18.7k times · Source

I have a staging site which I use to draft new features, changes and content to my actual website.

I don't want this to get indexed, but I'm hoping for a solution a little easier than having to add the below to every page on my site:

<meta name="robots" content="noindex, nofollow">

Can I do this in a way similar to how I added a password to the domain using a .htaccess file?

Answer

wutz picture wutz · Jan 29, 2012

The robots.txt standard is meant for this. Example

User-agent: *
Disallow: /protected-directory/

Search engines will obey this, but of course the content will still be published (and probably more easily discoverable if you put the URL in the robots.txt), so password protection via .htaccess is an option, too.

Related questions

How do I remove a single file from the staging area (undo git add)?

Situation: I have a Git repository with files already in the index. I make changes to several files, open Git and add these files to my staging area with "git add ." Question: How do I remove one of those files …

Read More
git add only modified changes and ignore untracked files

I ran "git status" and listed below are some files that were modified/or under the heading "changes not staged for commit". It also listed some untracked files that I want to ignore (I have a ".gitignore" file in these …

Read More
How do you move a commit to the staging area in git?

If you want to move a commit to the staging area - that is uncommit it and move all of the changes which were in it into the staging area (effectively putting the branch in the state that it would …

Read More