Robots.txt syntax not understood

robots.txt google-webmaster-tools
Lex Eichner picture Lex Eichner · Jul 10, 2012 · Viewed 7.6k times · Source

I submitted my robots.txt file ages ago to Google and it is still giving me a syntax not understood for the first line.

After Googling the most common problem is Google adding a '?' at the start of the line but it isnt doing that to me.

the url to the robots.txt is

www.leisurepursuits.co.uk/robots.txt

The error is:

Line 1: User-agent: * Syntax not understood

Answer

Jessica Eldridge picture Jessica Eldridge · Jan 18, 2013

This error shows because the expected robots.txt file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF.

If a character encoding is not a subset of UTF-8, then your robots.txt files is being parsed incorrectly.

First check your robots.txt url in http://www.asymptoticdesign.co.uk/cgi-bin/check-url.pl by selecting the second option “view source” and see that it responds successfully or not.

Now upload your robots.txt file again. It will not show error as shown previously.

Related questions

Sitemap contains urls which are blocked by robots.txt

We've had an accidental situation where our Wordpress site had the robots.txt set to disallow crawling for about 7 days. I'm now trying to play clean up and Webmaster Tools is saying "Sitemap contains urls which are blocked by robots.…

Read More
Can a relative sitemap url be used in a robots.txt?

In robots.txt can I write the following relative URL for the sitemap file? sitemap: /sitemap.ashx Or do I have to use the complete (absolute) URL for the sitemap file, like: sitemap: http://subdomain.domain.com/sitemap.ashx Why …

Read More
How to configure robots.txt to allow everything?

My robots.txt in Google Webmaster Tools shows the following values: User-agent: * Allow: / What does it mean? I don't have enough knowledge about it, so looking for your help. I want to allow all robots to crawl my website, is …

Read More