Sitemap contains urls which are blocked by robots.txt

wordpress search-engine sitemap robots.txt google-webmaster-tools
Rocco The Taco picture Rocco The Taco · Sep 26, 2014 · Viewed 10k times · Source

We've had an accidental situation where our Wordpress site had the robots.txt set to disallow crawling for about 7 days. I'm now trying to play clean up and Webmaster Tools is saying "Sitemap contains urls which are blocked by robots.txt." AFTER I"ve adjusted the robots.txt and allowed crawling. There is no reason why the URLS would still be blocked and when I visit the examples they appear OK.

Robots.txt URL: http://bit.ly/1u2Qlbx

Sitemap URL: http://bit.ly/1BfkSmx

My URLs where the robots.txt is blocking according to Webmaster Tools: http://bit.ly/1uLBRea OR http://bit.ly/1CsrHnr

Answer

Jérôme Verstrynge picture Jérôme Verstrynge · Sep 29, 2014

Don't worry, if you enabled the URLs again, Google's web crawlers will notice and crawl your pages again. The message will disappear from GWT within days.

Eventually, you may test your robots.txt with GWT's corresponding feature.

Related questions

Access denied for user 'root@localhost' (using password:NO)

I'm new to MySQL, I'm trying to run WordPress in my Windows desktop and it needs MySQL. I install everything with Web Platform Installer which is provided by Microsoft. I never set a root password for MySQL and in the …

Read More
Create a folder if it doesn't already exist

I've run into a few cases with WordPress installs with Bluehost where I've encountered errors with my WordPress theme because the uploads folder wp-content/uploads was not present. Apparently the Bluehost cPanel WordPress installer does not create this folder, though …

Read More
Correct file permissions for WordPress

I've had a look over here but didn't find any details on the best file permissions. I also took a look at some of WordPress's form's questions over here too but anybody that suggests 777 obviously needs a little lesson in …

Read More