How to prevent Googlebot from overwhelming site?

performance googlebot
lo_fye picture lo_fye · Aug 25, 2009 · Viewed 8.8k times · Source

I'm running a site with a lot of content, but little traffic, on a middle-of-the-road dedicated server.

Occasionally, Googlebot will stampede us, resulting in Apache maxing out its memory, and causing the server to crash.

How can I avoid this?

Answer

Gavin Miller picture Gavin Miller · Aug 25, 2009

You can set how your site is crawled using google's webmaster tools. Specifically take a look at this page: Changing Google's crawl rate

You can also restrict the pages that the google bot searches using a robots.txt file. There is a setting available for crawl-delay, but it appears that it is not honored by google.

Related questions

Which "href" value should I use for JavaScript links, "#" or "javascript:void(0)"?

The following are two methods of building a link that has the sole purpose of running JavaScript code. Which is better, in terms of functionality, page load speed, validation purposes, etc.? or function myJsFunc() { alert("myJsFunc"); } <a href="javascript:…

Read More
Fastest way to check if a value exists in a list

What is the fastest way to know if a value exists in a list (a list with millions of values in it) and what its index is? I know that all values in the list are unique as in this …

Read More
How to measure elapsed time in Python?

What I want is to start counting time somewhere in my code and then get the passed time, to measure the time it took to execute few function. I think I'm using the timeit module wrong, but the docs are …

Read More