How do I diagnose this PHP segmentation fault?

php segmentation-fault pear mail-queue
Dave picture Dave · Oct 26, 2012 · Viewed 65.1k times · Source

I am running a command-line PHP job that is resulting in a Segmentation Fault. This job has worked for a long time, but it processes something that is emailed in. Apparently there's something in this email that's break it, but I have no idea what. If I put the core file in gdb, it's really no help:

$ gdb /usr/local/bin/php core.20381 
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/php...done.
[New Thread 20381]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libmysqlclient_r.so.15...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libmysqlclient_r.so.15
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /usr/lib/libmcrypt.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libmcrypt.so.4
Reading symbols from /usr/lib/libltdl.so.7...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libltdl.so.7
Reading symbols from /lib/libpng12.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpng12.so.0
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libcurl-gnutls.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcurl-gnutls.so.4
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcrypt.so.11
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libgpg-error.so.0
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Can't read symbols from system-supplied DSO at 0x7fffb4bfc000: File truncated
Core was generated by `/usr/local/bin/php /src/prod/current/utils/cron.php --run=/utils/proces'.
Program terminated with signal 11, Segmentation fault.
#0  match (
    eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc3aa "N", 
    mstart=0x13708600 "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1, maximum-scale=1\">\n\n</head>\n\n<body style=\"padding"..., markptr=0x0, offset_top=4, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17680) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:470
470 {

If I do a 'bt', I just get this "match" statement over and over again. Here are the first few:

#1  0x000000000046ef1e in match (
    eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>, 
    offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17679) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803
#2  0x0000000000479789 in match (
    eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc5ca "V\002%\033U\002.", mstart=<value optimized out>, 
    markptr=<value optimized out>, offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=<value optimized out>)
    at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:1623
#3  0x000000000046ef1e in match (
    eptr=0x1370a886 "</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>, 
    offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17677) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803
#4  0x0000000000479789 in match (
    eptr=0x1370a886 "</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t"..., ecode=0x133dc5ca "V\002%\033U\002.", mstart=<value optimized out>, 
    markptr=<value optimized out>, offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=<value optimized out>)
    at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:1623
#5  0x000000000046ef1e in match (
    eptr=0x1370a885 "\t</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>, 
    offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17675) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803

I'm really at a loss here. I've tried running the command in strace, and it seems to be seg-faulting in the PEAR Mail_Queue library. Here's the end of the strace (some identifying information has been removed):

lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver/mysql.php", {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
open("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver/mysql.php", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
mmap(NULL, 55723, PROT_READ, MAP_SHARED, 6, 0) = 0x2b49f7bb4000
brk(0x987b000)                          = 0x987b000
brk(0x98bb000)                          = 0x98bb000
munmap(0x2b49f7bb4000, 55723)           = 0
close(6)                                = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6
fcntl(6, F_SETFL, O_RDONLY)             = 0
fcntl(6, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(6, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
connect(6, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("IP ADDRESS TO DB SERVER REMOVED")}, 16) = -1 EINPROGRESS (Operation now in progress)
fcntl(6, F_SETFL, O_RDWR)               = 0
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}])
setsockopt(6, SOL_SOCKET, SO_RCVTIMEO, "\2003\341\1\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
setsockopt(6, SOL_SOCKET, SO_SNDTIMEO, "\2003\341\1\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
setsockopt(6, SOL_IP, IP_TOS, [8], 4)   = 0
setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(6, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
poll([{fd=6, events=POLLIN}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}])
read(6, "C\0\0\0\n5.1.37-1ubuntu5.5-log\0Q\257\3\0009"..., 16384) = 71
write(6, "=\0\0\1\205\242\2\0\0\0\0@\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65) = 65
read(6, "\7\0\0\2\0\0\0\2\0\0\0", 16384) = 11
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(6, "\10\0\0\0\2xxxxx", 12)      = 12
read(6, "\7\0\0\1\0\0\0\2\0\0\0", 16384) = 11
lstat("/src/prod/current/www/application/pear/Mail/mime.php", {st_mode=S_IFREG|0770, st_size=50252, ...}) = 0
lstat("/src/prod/current/www/application/pear/Mail", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/current/www/application/pear", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/current/www/application", {st_mode=S_IFDIR|0776, st_size=4096, ...}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Segmentation fault

If anyone has any ideas or pointers to lines of investigation, I'd truly appreciate it. I'm really stumped on this one.

Answer

Trendfischer picture Trendfischer · Nov 7, 2012

I usually use xdebug if I encounter a segfault. I prefer to use the following line:

php -d xdebug.auto_trace=ON -d xdebug.trace_output_dir=mytracedir/ myscript.php

With the trace-output under mytracedir/, you can easily figure out the approximate php line, which breaks the script. Then rewrite the line in one statement per line and use error_log() for debugging vars. If you use __destruct() or __toString() it could get a bit more tricky there.

See here for more informations.

(If the extension is not loaded, but installed, try to add php -dzend_extension=xdebug.so [...])

Related questions

PHPUnit Segmentation fault

When a PHPUnit test fails normally on my dev box (Linux Mint), it causes a "Segmentation Fault" on my Continous Integration box (Centos). Both machines are running the same version of PHPUnit. My dev box is running PHP 5.3.2-1ubuntu4.9, …

Read More
NGINX + PHP5-FPM segfaults under high load

I have been dealing with this problem all day and it is driving me insane. All Google results and searches here lead to dead ends. I hope someone can work with me to provide a solution for myself and future …

Read More
what is root cause of 'child pid 10708 exit signal Segmentation fault (11)' error?

I am getting more and more child pid 10708 exit signal Segmentation fault (11) errors. what is the root cause of it and how to fix it ? Is php ini memory is associated with this? I am using apache2 server with php. …

Read More