Wrapper ping timeout and mule trapped

Question

As stated in the title. After that, wrapper failed to start another JVM, and my mule service is stoped. and clues on this?

There's a little load on the server but the load should not be too much.

Below is the related log:

Send a packet PING : ok
read a packet PING : ok
Got ping response from JVM
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
JVM appears hung: Timed out waiting for signal from JVM.
JVM did not exit on request, terminated server listening on port 32002.
Waiting 5 seconds before launching another JVM.
Signal trapped. Details:
signal number=17 (SIGCHLD), source="unknown"
Received SIGCHLD, checking JVM process status.
JVM exited in response to signal SIGKILL (9).
JVM process exited with a code of 1, however the wrapper exit code was already 1.
Unable to start a JVM
<-- Wrapper Stopped

here's the log when I restart mule useing:mule start.

Working directory set to: /root
Spawning intermediate process...
Spawning daemon process...
--> Wrapper Started as Daemon
Using tick timer.
server listening on port 32000.
Classpath element, wrapper.java.classpath.1, does not exist: %MULE_LIB%
Command[0] : java
Command[1] : -Dmule.home=/opt/skybus
Command[2] : -Dmule.base=/opt/skybus
Command[3] : -Djava.net.preferIPv4Stack=TRUE
Command[4] : -XX:MaxPermSize=128m
Command[5] : -Djava.endorsed.dirs=/opt/skybus/lib/endorsed
Command[6] : -Xms2048m
Command[7] : -Xmx2048m
Command[8] : -Djava.library.path=%LD_LIBRARY_PATH%:/opt/skybus/lib/boot
Command[9] : -classpath
Command[10] : %MULE_LIB%:/opt/skybus/conf:/opt/skybus/lib/boot/mule-module-boot-        3.2.1.jar:/opt/skybus/lib/boot/log4j-1.2.14.jar:/opt/skybus/lib/boot/mule-module-reboot-3.2.1.jar:/opt/skybus/lib/boot/wrapper-3.2.3.jar:/opt/skybus/lib/boot/commons-cli-1.2.jar:/opt/skybus/lib/boot/mule-module-logging-3.2.1.jar
Command[11] : -Dwrapper.key=XmncggXh3D4jnC8q
Command[12] : -Dwrapper.port=32000
Command[13] : -Dwrapper.jvm.port.min=31000
Command[14] : -Dwrapper.jvm.port.max=31999
Command[15] : -Dwrapper.debug=TRUE
Command[16] : -Dwrapper.pid=30321
Command[17] : -Dwrapper.version=3.2.3
Command[18] : -Dwrapper.native_library=wrapper
Command[19] : -Dwrapper.service=TRUE
Command[20] : -Dwrapper.cpu.timeout=10
Command[21] : -Dwrapper.jvmid=1
Command[22] : org.mule.module.reboot.MuleContainerBootstrap
Command[23] : start0
Launching a JVM...
Starting the Mule Container...
WrapperManager class initialized by thread: main  Using classloader: sun.misc.Launcher$AppClassLoader@192d342
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

Wrapper Manager: JVM #1
Running a 32-bit JVM.
Wrapper Manager: Registering shutdown hook
Wrapper Manager: Using wrapper
Load native library.  One or more attempts may fail if platform specific libraries do not exist.
Loaded native library: libwrapper-linux-x86-32.so
Calling native initialization method.
Inside native WrapperManager initialization method
Java Version   : 1.6.0_10-beta-b25 Java HotSpot(TM) Server VM
Java VM Vendor : Sun Microsystems Inc.

WrapperManager.start(org.mule.module.reboot.MuleContainerWrapper@192b996, args["start0"]) called by thread: main
Control event monitor thread started.
Startup runner thread started.
Communications runner thread started.
Open socket to wrapper...Wrapper-Connection
Opened Socket from 31000 to 32000
Send a packet KEY : XmncggXh3D4jnC8q
handleSocket(Socket[addr=/127.0.0.1,port=32000,localport=31000])
accepted a socket from 127.0.0.1 on port 31000
read a packet KEY : XmncggXh3D4jnC8q
Got key from JVM: XmncggXh3D4jnC8q
send a packet LOW_LOG_LEVEL : 1
send a packet PING_TIMEOUT : 30
send a packet PROPERTIES : (Property Values)
Start Application.
send a packet START : start
Received a packet LOW_LOG_LEVEL : 1
Wrapper Manager: LowLogLevel from Wrapper is 1
Received a packet PING_TIMEOUT : 30
PingTimeout from Wrapper is 30000
Received a packet PROPERTIES : (Property Values)
Received a packet START : start
calling WrapperListener.start()
Waiting for WrapperListener.start runner thread to complete.
WrapperListener.start runner thread started.
2012-04-01 14:53:42,297 INFO [org.mule.module.launcher.MuleContainer] - <
**********************************************************************
* Mule ESB and Integration Platform                                  *
* Version: 3.2.1 Build: 23380                                        *
* MuleSoft, Inc.                                                     *
* For more information go to http://www.mulesoft.org                 *
*                                                                    *
* Server started: 12-4-1 下午2:53                                    *
* JDK: 1.6.0_10-beta (mixed mode)                                    *
* OS: Linux (2.6.18-92.el5xen, i386)                                 *
* Host: localhost.localdomain (10.123.75.149)                        *
**********************************************************************>

Answer 1

from the first log file, I can see that the JVM was unresponsive to the ping requests of the Wrapper:

send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping
send a packet PING : ping

so the Wrapper assumes the JVM is frozen and therefore tries to stop and restart it. But in fact it appears that due to some heavy load on, the machine the Wrapper couldn't even start another JVM instance. You could extend the wrapper.ping.timeout, which would give the JVM more time on ping requests, if heavy load is happening frequently on your machine.

http://wrapper.tanukisoftware.com/doc/english/prop-ping-timeout.html

Hope this helps you out.

Cheers