Reaching MaxRequestWorkers limits caused by bot traffic ERR_CONNECTION_TIMED_OUT

Question

I had an ERR_CONNECTION_TIMED_OUT and the website went down for a few minutes several times in a time-frame of approximately 1.5 hours, and this is something I found in the logs:

[Mon May 21 11:21:51.236380 2018] [mpm_worker:error] [pid 3206:tid 140035442734330] AH00287: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

I went to /etc/apache2/conf/httpd.conf and found this:

MaxRequestWorkers [The number specified for my server]

I just changed server yesterday and this is one theory of the networking guys about what could be the cause of the problem:

The server is hitting maxworkers due to the type of traffic that are coming in, which are bot traffic. BingBot and Googlebot are crawling your site again, likely because of the new server. You can manage the crawl rate of these bots so that they do request so much at any given time.

Is this a plausible explanation? I find this hard to believe.

Edit 1: It sounds like a denial-of-service (DDoS) attack performed by Google!

Answer 1

There are even articles that talk about how bot traffic can affect a website:

• http://support.hostgator.com/articles/specialized-help/telling-google-how-often-to-crawl-your-website
• https://support.google.com/webmasters/answer/48620?hl=en
• http://support.hostgator.com/articles/specialized-help/telling-bing-how-often-to-crawl-your-website
• http://support.hostgator.com/articles/hosting-guide/lets-get-started/how-to-use-robots-txt
• http://support.hostgator.com/articles/specialized-help/technical/apache-htaccess/user-agent-blocks-mainly-for-bots

I am not sure if bot traffic was really the cause of my problem, but the error message included this part: consider raising the MaxRequestWorkers setting. That is what I did and everything seems to be working correctly now.

IMPORTANT: If you do this, do not forget to read https://httpd.apache.org/docs/current/mod/mpm_common.html and remember that if you increase MaxRequestWorkers, you will have to increase ServerLimit accordingly. The documentation explains that.