Meta tag vs robots.txt

seo robots.txt meta-tags
keruilin picture keruilin · Jul 27, 2010 · Viewed 17.5k times · Source

  1. Is it better to use meta tags* or the robots.txt file for informing spiders/crawlers to include or exclude a page?

  2. Are there any issues in using both the meta tags and the robots.txt?

*Eg: <#META name="robots" content="index, follow">

Answer

user2696762 picture user2696762 · Aug 19, 2013

There is one significant difference. According to Google they will still index a page behind a robots.txt DENY, if the page is linked to via another site.

However, they will not if they see a metatag:

While Google won't crawl or index the content blocked by robots.txt, we might still find and index a disallowed URL from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the site can still appear in Google search results. You can stop your URL from appearing in Google Search results completely by using other URL blocking methods, such as password-protecting the files on your server or using the noindex meta tag or response header.

Related questions

Can a relative sitemap url be used in a robots.txt?

In robots.txt can I write the following relative URL for the sitemap file? sitemap: /sitemap.ashx Or do I have to use the complete (absolute) URL for the sitemap file, like: sitemap: http://subdomain.domain.com/sitemap.ashx Why …

Read More
Ignore urls in robot.txt with specific parameters?

I would like for google to ignore urls like this: http://www.mydomain.com/new-printers?dir=asc&order=price&p=3 All urls that have the parameters dir, order and price should be ignored but I dont have experience …

Read More
How to allow crawlers access to index.php only, using robots.txt?

If i want to only allow crawlers to access index.php, will this work? User-agent: * Disallow: / Allow: /index.php

Read More