Tomcat doesn't stop. How can I debug this?

Question

I have a Tomcat 7 running in Linux that I start via $CATALINA_HOME/bin/startup.sh and shutdown via $CATALINA_HOME/bin/shutdown.sh
from /etc/init.d

All is ok except 1 problem. Sometimes tomcat does not stop.
Although I stop it and I see in catalina.out logs that is going down, if I do ps -ef I can still see the process running.

What could be the problem? How can I debug this? My feeling is, that this is related to threads.

So the parts that are suspicious are the following:
1) I use Log4j's LogManager to detect if the log4j configuration has been changed, but I do Log4jManager.shutdown on a contextDestroyed ServletContextListener
2) I use H2 database and I see on shutdown:

SEVERE: The web application [/MyApplication] appears to have started a
thread named [H2 Log Writer MYAPPLICATION] but has failed to stop it.
This is very likely to create a memory leak

SEVERE: The web application [/MyApplication] appears to have started a
thread named [H2 File Lock Watchdog
/opt/myOrg/tomcat/webapps/MyApplication/db/myDatabase.lock.db] but has
failed to stop it. This is very likely to create a memory leak. Apr 2,
2012 9:08:08 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads SEVERE: The web application [/MyApplication]
appears to have started a thread named [FileWatchdog] but has failed
to stop it. This is very likely to create a memory leak.

Any help please? How can I detect the problem here?

UPDATE:
I did a kill -3 as suggested by @daveb, and in the catalina.out I see:

JVMDUMP006I Processing dump event "user", detail "" - please wait. JVMDUMP032I JVM requested Java dump using '/etc/init.d/javacore.20120402.093922.2568.0001.txt' in response to an event JVMDUMP010I Java dump written to /etc/init.d/javacore.20120402.093922.2568.0001.txt JVMDUMP013I Processed dump event "user", detail "".

There is a javacore in /etc/init.d but I don't know how to process it. I.e. what parts should I investigate

Answer 1

Find out what threads are still running (or blocked, waiting to run) by using jstack or sending a signal to the process:

kill -3 pid

When you know this, you can make whatever it was that started them hook into the shutdown notification to stop the threads. Or make those threads deamon threads.

See This tomcat shutdown question for more details on this.

If you don't know where your threads were created, then consider adding names to them - executors can take thread factories, and you can use those factories to set the deamon status of a thread and also to name it - so your stack trace will be clearer.