How do I prevent Bing from swamping my site with traffic irregularly?

web-crawler robots.txt bing bingbot

Tim Haines · Feb 8, 2011 · Viewed 10.8k times · Source

Bingbot will hit my site pretty hard for a couple of hours each day, and will be extremely light for the rest of the time.

I'd either like to smooth out its crawls, reduce its rate limit, or block it altogether. It doesn't really send through any real visitors.

Is there a way I can smooth its crawling, or rate limit it?

Answer

Their webmaster blog says that they support adding a crawl-delay parameter to your robots.txt file to throttle the bingbot.

User-agent: msnbot
Crawl-delay: 1

There's a bit more explanation in the webmaster FAQ PDF