robots.txt; What encoding?

search robots.txt
user188962 picture user188962 · Sep 28, 2010 · Viewed 7.4k times · Source

I am about to create a robots.txt file.

I am using notepad.

How should I save the file? UTF8, ANSI or what?

Also, should it be a capital R?

And in the file, I am specifying a sitemap location. Should this be with a capital S?

  User-agent: *
  Sitemap: http://www.domain.se/sitemap.xml

Thanks

Answer

Roland Illig picture Roland Illig · Sep 28, 2010

Since the file should consist of only ASCII characters, it normally doesn't matter if you save it as ANSI or UTF-8.

However, you should choose ANSI if you have a choice because when you save a file as UTF-8, notepad adds the Unicode Byte Order Mark to the front of the file, which may make the file unreadable for interpreters that only know ASCII.

Related questions

how to restrict the site from being indexed

I know this question was being asked many times but I want to be more specific. I have a development domain and moved the site there to a subfolder. Let's say from: http://www.example.com/ To: http://www.example.…

Read More
How can i fix "Googlebot can't access your site" issue?

I just keep getting a message about "Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall …

Read More
grep a file, but show several surrounding lines?

I would like to grep for a string, but also show the preceding five lines and the following five lines as well as the matched line. How would I be able to do this?

Read More