Is it possible to list multiple user-agents in one line?

Question

Is it possible in robots.txt to give one instruction to multiple bots without repeatedly having to mention it?

Example:

User-agent: googlebot yahoobot microsoftbot
Disallow: /boringstuff/

Answer 1

It's actually pretty hard to give a definitive answer to this, as there isn't a very well-defined standard for robots.txt, and a lot of the documentation out there is vague or contradictory.

The description of the format understood by Google's bots is quite comprehensive, and includes this slightly garbled sentence:

Muiltiple start-of-group lines directly after each other will follow the group-member records following the final start-of-group line.

Which seems to be groping at something shown in the following example:

user-agent: e
user-agent: f
disallow: /g

According to the explanation below it, this constitutes a single "group", disallowing the same URL for two different User Agents.

So the correct syntax for what you want (with regards to any bot working the same way as Google's) would then be:

User-agent: googlebot
User-agent: yahoobot
User-agent: microsoftbot
Disallow: /boringstuff/

However, as Jim Mischel points out, there is no point in a robots.txt file which some bots will interpret correctly, but others may choke on, so it may be best to go with the "lowest common denominator" of repeating the blocks, perhaps by dynamically generating the file with a simple "recipe" and update script.