I get "Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler" error in tomcat

Question

I have tomcat web app on VPS and the tomcat sometimes (about once a month) crashes with the following error in catalina.out:

Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated.

Here are some details about my configuration:

• VPS: debian-5.0-x86_64

• RAM: 2.5 gb,

• virtual processors: 8

• HDD: 60gb hdd - 70% free

• Tomcat 7.0

• java -version:

  java version "1.6.0_18"
  OpenJDK Runtime Environment (IcedTea6 1.8.13) (6b18-1.8.13-0+squeeze1)
  OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)
  

• Java params: -Xms512m -Xmx1024m

I have also Apache-PHP on that server.

I'm monitoring server load with Munin and it shows me that memory and CPU usage is always stable and there were no any increases before crash.

I'm also logging java memory usage via java.lang.Runtime class, and it shows that jvm always uses max200Mb memory and there were no increase before crash. The last log before crash was 40 seconds ago and that time used memory was: 152Mb.

My web app also runs 6-7 threads that collecting data from different public APIs. These threads start when tomcat is starting, and they always running with periodic sleeps.

Can you please tell me why it crashes? How can I find the reason?

Answer 1

Lets unpick this:

Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated.

First of all, it looks like something sent the JVM (Tomcat) process a SIGTERM signal. It has to be something external to the JVM that did that. A JVM doesn't send signals to itself¹.

So you need to figure out what is doing that. My guess first guess would be the OOM killer ... but the OOM killer uses SIGKILL not SIGTERM. And the JVM never sees the SIGKILL coming!

(You can confirm that it is not the OOM killer by looking in "/var/log/messages" ... or wherever your system logs kernel messages. See How to Configure the Linux Out-of-Memory Killer)

If it is not the OOM killer, then there are a few ways to find the source of a signal:

• Finding the source of signals on Linux with strace, auditd, or systemtap

And once you have the source of the signal, you will have clues as to why it was sent.

---

The other noteworthy thing is that the OutOfMemoryError occurred while handling the SIGTERM. This strongly suggests (to me) that the root cause is that something has detected that Tomcat is using too much memory, and has sent it a SIGTERM to make it go away (cleanly). I surmise that what happens then is that the JVM goes to the OS to ask for a little bit more memory (to handle the SIGTERM) and the OS says "No", and the JVM throws an OutOfMemoryError. Unfortunately, the JVM is now in a state where it cannot either exit cleanly or recover. Hence it says "the VM may need to be forcibly terminated".

---

Anyhow. This looks to me like a rather unusual manifestation of a common Java problem. You most likely have a bug in the webapps running in your Tomcat that is leaking memory. If that is the case, the only real solution is to find and fix the bug. (Increasing the heap size ... if possible ... only puts the problem off. It may decrease the interval between crashes, but it is unlikely to prevent them.)

Assuming that you are ready to bite the bullet:

• How to find a Java Memory Leak

---

^{1 - Unless something is doing something nutty in native code ...}