Robots.txt for multiple domains

Question

We have different domains for each language

1. www.abc.com
2. www.abc.se
3. www.abc.de

And then we have different sitemap.xml for each site. In robots.txt, I want to add sitemap reference for each domain.

1. Is it possible to have multiple sitemap references for each domain in single robots.txt?
2. If there are multiple, which one does it pick?

Answer 1

I'm using the following solution in .htaccess after all domain redirects and www to non-www redirection.

# Rewrite URL for robots.txt
RewriteRule ^robots\.txt$ robots/%{HTTP_HOST}.txt [L]

Create a new directory in your root called robots. Create a text file filled with the specific robots information for every domain.

• /robots/abc.com.txt
• /robots/abc.se.txt
• /robots/abc.de.txt