APC and child pid XXXXX exit signal Segmentation fault

php linux apache2 apc xcache
Soroush Falahati picture Soroush Falahati · Mar 28, 2013 · Viewed 12.3k times · Source

First I had xCache installed on server, I had xCache on lot of my servers but in this one after couple of days you go nothing except a blank page with this error in apache error log:

 child pid XXXXX exit signal Segmentation fault

As far as I know this mean some sort of memory corruption.

So I removed xCache from php.ini and installed APC. But things are worse now. After one hour or less/more server become unresponsive. I can clearly see in my website's server-status that there are lot of connections in W mode. And they dont have any thing in common. Some of them are from Joomla (first page) and some from vBulletin (forums) and some other are about other custom codes. If I let it for more time I will going to have this error in Apache error log:

 Server reached MaxClients setting, consider raising the MaxClients setting

Apache is in mod_php and mod_ruid2 mode.

Here is result of server-status when hanged:

Server uptime: 52 minutes 57 seconds
Total accesses: 31025 - Total Traffic: 279.0 MB
CPU Usage: u418.14 s64.28 cu0 cs0 - 15.2% CPU load
9.77 requests/sec - 89.9 kB/second - 9.2 kB/request
398 requests currently being processed, 27 idle workers

WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WW___WW_WW_W_WWW__WWWW_W_.......................................
_W___W___WW_W_W_W_W______.......................................

Here is error log for xCache (lot of these lines):

[Thu Mar 28 09:50:38 2013] [notice] child pid 1607 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:51:57 2013] [notice] child pid 1643 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:52:32 2013] [notice] child pid 1724 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:55:21 2013] [notice] child pid 1672 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:55:56 2013] [notice] child pid 1471 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:56:49 2013] [notice] child pid 1753 exit signal Aborted (6)
[Thu Mar 28 09:58:29 2013] [notice] child pid 1810 exit signal Segmentation fault (11)
[Thu Mar 28 09:58:29 2013] [notice] child pid 1877 exit signal Segmentation fault (11)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 10:01:34 2013] [notice] child pid 1844 exit signal Segmentation fault (11)
[Thu Mar 28 10:01:34 2013] [notice] child pid 1913 exit signal Aborted (6)
[Thu Mar 28 10:01:34 2013] [notice] child pid 1941 exit signal Segmentation fault (11)
[Thu Mar 28 10:01:36 2013] [notice] child pid 2007 exit signal Segmentation fault (11)
[Thu Mar 28 10:01:37 2013] [notice] child pid 2035 exit signal Segmentation fault (11)
[Thu Mar 28 10:01:37 2013] [notice] child pid 2036 exit signal Segmentation fault (11)

Error log for APC before reaching max client (from one restart to other restart):

[Thu Mar 28 19:17:10 2013] [notice] Apache/2.2.24 (Unix) mod_ssl/2.2.24 OpenSSL/1.0.0-fips DAV/2 PHP/5.2.17 configured -- resuming normal operations
[Thu Mar 28 19:21:40 2013] [notice] child pid 1356 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:41 2013] [notice] child pid 1520 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:42 2013] [notice] child pid 1330 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:42 2013] [notice] child pid 1518 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:46 2013] [notice] child pid 1967 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:46 2013] [notice] child pid 1968 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:49 2013] [notice] child pid 1939 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:50 2013] [notice] child pid 2034 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:51 2013] [notice] child pid 2062 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:57 2013] [notice] child pid 2091 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:58 2013] [notice] child pid 2090 exit signal Segmentation fault (11)
[Thu Mar 28 19:25:34 2013] [notice] caught SIGTERM, shutting down
[Thu Mar 28 19:25:37 2013] [notice] SSL FIPS mode disabled
[Thu Mar 28 19:25:37 2013] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Mar 28 19:25:37 2013] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Mar 28 19:25:37 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 28 19:25:38 2013] [notice] SSL FIPS mode disabled
[Thu Mar 28 19:25:38 2013] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Mar 28 19:25:38 2013] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Mar 28 19:25:38 2013] [notice] Apache/2.2.24 (Unix) mod_ssl/2.2.24 OpenSSL/1.0.0-fips DAV/2 PHP/5.2.17 configured -- resuming normal operations

Little from PHP Info:

System  Linux xxx.xxxxx.com 2.6.32-358.0.1.el6.x86_64 #1 SMP Wed Feb 27 06:06:45 UTC 2013 x86_64
Build Date  Mar 17 2013 12:55:14
Configure Command   './configure' '--with-apxs2' '--with-curl=/usr/local/lib' '--with-gd' '--with-ttf' '--with-gettext' '--with-jpeg-dir=/usr/local/lib' '--with-freetype-dir=/usr/local/lib' '--with-kerberos' '--with-openssl' '--with-mcrypt' '--with-mhash' '--with-mysql=/usr' '--with-mysqli=/usr/bin/mysql_config' '--with-pdo-mysql=/usr' '--with-pcre-regex=/usr/local' '--with-pear' '--with-png-dir=/usr/local/lib' '--with-xsl' '--with-zlib' '--with-zlib-dir=/usr/local/lib' '--with-iconv=/usr/local' '--enable-bcmath' '--enable-calendar' '--enable-exif' '--enable-ftp' '--enable-gd-native-ttf' '--enable-magic-quotes' '--enable-safe-mode' '--enable-soap' '--enable-sockets' '--enable-mbstring' '--enable-zip' '--enable-wddx'
Server API  Apache 2.0 Handler
Virtual Directory Support   enabled
Configuration File (php.ini) Path   /usr/local/lib
Loaded Configuration File   /usr/local/lib/php.ini
Scan this dir for additional .ini files     (none)
additional .ini files parsed    (none)
PHP API     20041225
PHP Extension   20060613
Zend Extension  220060519
Debug Build     no
Thread Safety   enabled
Zend Memory Manager     enabled
IPv6 Support    enabled
Registered PHP Streams  https, ftps, compress.zlib, php, file, data, http, ftp, zip
Registered Stream Socket Transports     tcp, udp, unix, udg, ssl, sslv3, sslv2, tls
Registered Stream Filters   zlib.*, convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed

APC settings:

enabled
Version     3.1.13
APC Debugging   Disabled
MMAP Support    Enabled
MMAP File Mask  no value
Locking type    pthread read/write Locks
Serialization Support   php
Revision    $Revision: 327136 $
Build Date  Mar 28 2013 12:40:18

Php -v:

PHP 5.2.17 (cli) (built: Mar 17 2013 12:55:24) 
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies

httpd -V:

Server version: Apache/2.2.24 (Unix)
Server built:   Mar 10 2013 09:18:00
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.4.6, APR-Util 1.4.1
Compiled using: APR 1.4.6, APR-Util 1.4.1
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

Any Idea?! I will reward 50 :)

Answer

user149341 picture user149341 · Mar 31, 2013

From the looks of your server status page, you are trying to run PHP as mod_php with a threaded Apache MPM. Many PHP extensions, likely including APC and Xcache, are not thread-safe. As such, they will crash frequently under load.

You will need to switch Apache to the prefork MPM, or run PHP as CGI or FastCGI.

Related questions

php.ini changes but not effective in ubuntu

I want change the limit of php upload file's size And this is some information of my phpinfo. Configuration File (php.ini) Path /etc/php5/apache2 Loaded Configuration File /etc/php5/apache2/php.ini And this is the content of …

Read More
VirtualHost is not working under Apache 2.4.6 on CentOS 7

I'm trying to setup some VH in Apache 2.4.6 on CentOS 7 but without success since it's not working. This is what I've tried til now: Since in /etc/httpd/conf/httpd.conf is this line Include conf.modules.d/*.conf then …

Read More
Where can I find php.ini?

A few years ago I installed Apache 2.2x and PHP 5.3.1 on a Linux server I maintain. I used .tar.gz's and built them as instructed (instead of rpms and what-have-you). And all was fine. Today I need to install this …

Read More