How to interpret a Java thread stack?

Question

In line with this question: How to get stack trace of a thread

I am wondering if anyone could point to information about interpreting Java thread stacks extracted with jstack.

My situation is that I've got a Java EE 5 application running on GlassFish v2.1.1 that hangs periodically (at least 2 -3 times a day).To get it running again I have to kill the Glassfish process and start the domain again. The application usually gets slower and slower responding until it finally hangs completely. Once it has hung I can't get a thread stack. I've been able to get a thread stack as it gets slower and slower but I can't see to locate the problem or the threads that have hung.

Here is the latest thread dump: http://issues.i2m.dk/attachments/66/threadstack.201103211046.txt

Snippet of thread dump:

"p: thread-pool-1; w: 20" daemon prio=10 tid=0x00007efd18843800 nid=0x5f89 in Object.wait() [0x00007efd03381000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"TP-Processor83" daemon prio=10 tid=0x0000000043179000 nid=0x5f87 runnable [0x00007efd03583000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efe72696090> (a java.io.BufferedInputStream)
    at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)
    at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:662)

"p: thread-pool-1; w: 19" daemon prio=10 tid=0x00007efd18710800 nid=0x5f86 in Object.wait() [0x00007efd03684000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 18" daemon prio=10 tid=0x0000000042a5a000 nid=0x5f85 in Object.wait() [0x00007efd03785000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 17" daemon prio=10 tid=0x0000000042a57000 nid=0x5f84 in Object.wait() [0x00007efd03886000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 16" daemon prio=10 tid=0x000000004321b000 nid=0x5f83 in Object.wait() [0x00007efd03987000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 15" daemon prio=10 tid=0x0000000043219800 nid=0x5f82 in Object.wait() [0x00007efd03a88000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 14" daemon prio=10 tid=0x00000000429b6000 nid=0x5f81 in Object.wait() [0x00007efd03b89000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"TP-Processor82" daemon prio=10 tid=0x0000000043896800 nid=0x5f80 runnable [0x00007efd03c8a000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efe72829510> (a java.io.BufferedInputStream)
    at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)
    at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:662)

"Thread-44980" daemon prio=10 tid=0x00007efd202c3800 nid=0x5d74 runnable [0x00007efd067b5000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efd78fc0058> (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

"Thread-44979" daemon prio=10 tid=0x00007efd20d76000 nid=0x5d73 runnable [0x00007efd062b0000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efd7789a4e0> (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

"TP-Monitor" daemon prio=10 tid=0x00007efd2139f800 nid=0x5b1 in Object.wait() [0x00007efd0d453000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d46d6c0> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:561)
    - locked <0x00007efd4d46d6c0> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:662)

"TP-Monitor" daemon prio=10 tid=0x00007efd20143000 nid=0x5b0 in Object.wait() [0x00007efd0d554000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d96dcd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:561)
    - locked <0x00007efd4d96dcd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:662)

"httpWorkerThread-4848-48" daemon prio=10 tid=0x00007efd18a82800 nid=0x5aa in Object.wait() [0x00007efd0db5a000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d09a620> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d09a620> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:120)


"GrizzlyPipelineStatWorkerThread-4848-2" daemon prio=10 tid=0x00007efd20033000 nid=0x579 waiting on condition [0x00007efd10c8b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d098f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"GrizzlyPipelineStatWorkerThread-4848-1" daemon prio=10 tid=0x00007efd2124f000 nid=0x578 waiting on condition [0x00007efd10d8c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d098f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"httpSSLWorkerThread-8181-48" daemon prio=10 tid=0x0000000042041800 nid=0x575 in Object.wait() [0x00007efd1108f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d0c8308> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d0c8308> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
    at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)


"SelectorThread-4848" prio=10 tid=0x00007efd207ba800 nid=0x546 runnable [0x00007efd13fbe000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4d9c24c0> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4d9c24a8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4d9c1d58> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4d9c27c0> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)


"httpSSLWorkerThread-8080-49" daemon prio=10 tid=0x00007efd20bfc800 nid=0x541 in Object.wait() [0x00007efd144c3000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d0f97a8> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d0f97a8> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
    at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)


"SelectorThread-8181" prio=10 tid=0x00007efd2161c000 nid=0x50e runnable [0x00007efd177f6000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4d3a6800> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4d3a67e8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4d3a6098> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4d3a6b00> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)

"GrizzlyPipelineStatWorkerThread-8080-1" daemon prio=10 tid=0x00007efd20f4c800 nid=0x50a waiting on condition [0x00007efd17bfa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d120680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"GrizzlyPipelineStatWorkerThread-8080-0" daemon prio=10 tid=0x00007efd20966800 nid=0x509 waiting on condition [0x00007efd17cfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d120680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"SelectorThread-8080" prio=10 tid=0x00007efd201a0800 nid=0x508 runnable [0x00007efd17dfc000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4dc827a0> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4dc82788> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4dc82088> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4dc82aa0> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]" daemon prio=10 tid=0x00007efd207af800 nid=0x503 waiting on condition [0x00007efd1c76c000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
    at java.lang.Thread.run(Thread.java:662)


"Timer-13" daemon prio=10 tid=0x00007efd202d2800 nid=0x4fe in Object.wait() [0x00007efd1c96e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4ab21998> (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x00007efd4ab21998> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"Timer-9" daemon prio=10 tid=0x00007efd209ad000 nid=0x4f9 in Object.wait() [0x00007efd1d7d1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4a82c750> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x00007efd4a82c750> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"Thread-11" daemon prio=10 tid=0x00007efd183f6800 nid=0x4c0 waiting on condition [0x00007efd1efe7000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd48273738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at com.sun.enterprise.management.support.LoaderRegThread.processFOREVER(LoaderRegThread.java:263)
    at com.sun.enterprise.management.support.LoaderRegThread.run(LoaderRegThread.java:311)


"SelectorThread" daemon prio=10 tid=0x00007efd183ff800 nid=0x4ba runnable [0x00007efd1f5ed000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd48171e70> (a sun.nio.ch.Util$1)
    - locked <0x00007efd48171e58> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd48171a98> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:283)

I'm looking for general information about interpreting the dump so that I can diagnose problems in the future. Thanks.

Answer 1

Couple of things that might help you

• Here are the different thread states explained like BLOCKED, WAITING and etc.
• Java will find dead locks for you and you will see output like described in Detecting Java Thread Deadlocks with 'jstack'.
• You probably have a live lock. Here you need information on your system and going through the dump and the source code.

How to move forward with current problem?

• Based on this dump I can tell you there are couple of objects that are locked and waited on from multiple threads. Look for strings '- waiting on' and '- locked'. I don't know the logic of your app to tell much more.