High CPU mod_fcgid: can't apply process slot
I get the following warning:
mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper
As soon as this warning appears server CPU raises to 97% - 101%
The server gets around 140K pageviews daily. This warning appears mostly around 11:00-12:00 Besides the 140K pageviews calculated by Analytics it also makes requests from third party websites that are displaying a text-ad on their website. Around 2700 websites are displaying one advertisement and are making a total of 22139 requests per hour together to the server.
I tried: chmod 755 /var/log/httpd but there isn't a folder "httpd" in /var/log
I tried to adjust MaxSpareServers and MaxClients to whatever possible high values. But doesn't effect at all. After Apache has restarted it seems to be fine for a while (server is still responding little bit slow and warnings are no longer appearing in the log file). But after a while the same warning appears and CPU raises again.
If someone has suggestions please let me know, thanks all.
Answer
This warning tells you that the FastCGI process pool is exhausted, and it has a global limit of FcgidMaxProcesses and a per-script limit of FcgidMaxProcessesPerClass (see http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxprocesses).
The size of this pool may be tuned with the following information :
- in the case where your application is CPU bound and you don't want it to slow down, you will set FcgidMaxProcesses to the number of CPU cores on your server; the server average load will tend to this number at peak hours. Your server will be serving clients without any slow down but will also reject some requests to keep up. Rejects are logged "can't apply process slot" on the server side and seen as 503 errors on the client side
- in the (common) case where your application is not totally CPU-bound, for instance it waits a faire amount its database server for replies (or other I/O or remote ressources), you may have FcgidMaxProcesses a bit higher than your number of CPU cores. As a rule of thumb I start with twice the number of CPU cores then check the peak average load for a whole day (a week is better). Then adjust until it peaks at avg. load = number of CPU cores.
In any case the default value of FcgidMaxProcesses (1000) is outrageous. You run a dumb 'ab -c100 -n10000
Just to be sure, let me state it another way : the "can't apply process slot" error is a normal error, because your server has a fixed amount of ressources, and those limits may be hit. If you push that limit to high wrt. to your server ressources, the warning might never appear again, but your server will crumble under the load.
And BTW, MaxSpareServers and MaxClients are not related to your problem, they regulate the Apache process pool, not the FastCGI process pool. which is another topic.