ActiveMQ won't restart because KahaDB is locked

java jetty activemq restart
IAmYourFaja picture IAmYourFaja · Oct 7, 2012 · Viewed 15.4k times · Source

Locally to start/stop ActiveMQ (5.6) on my dev machine I just run ./activemq start and ./activemq stop respectively.

On our QA machine we have it installed as a service and run service activemq start and service activemq stop respectively.

I just went to bounce the QA machine and issued service activemq stop, then service activemq start to restart it.

I see a process ID being created, and if I run ps -aef | grep activemq I see the living, breathing process of an ActiveMQ server.

But if I go to http://<qa-server>:8161/admin I get the typical error that you see when a server is down:

Firefox can't establish a connection to the server at :8161.

Edit: I have now tried both the ./active start and service activemq start methods, and both produce the same issue: I see a process being created, but nothing in the web admin tool.

I checked ActiveMQ's home directory and don't see any type of logs/ directory, so I'm not even sure where to begin debugging the issue.

Either AMQ is not restarting, or its web admin app isn't restarting or functioning properly; either way I have no idea where to start. Thanks in advance!

Edit:

I see the following error in data/activemq.log:

2012-10-07 11:37:14,501 | INFO | Database /qa-server/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/qa-server/kahadb/lock' could not be locked. | org.apache.activemq.store.kahadb.MessageDatabase | main 2012-10-07 11:37:24,504 | INFO | Database /qa-server/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/qa-server/kahadb/lock' could not be locked. | org.apache.activemq.store.kahadb.MessageDatabase | main

Answer

IAmYourFaja picture IAmYourFaja · Oct 9, 2012

Turns out there were multiple AMQ servers in our QA environment. When I shutdown the first server, an exception was thrown for some reason and so it didn't release the lock. Possession of that lock then went to the other AMQ instance (the first server was the master, the 2nd server was the slave).

When I tried restarting the first server (the master), it wouldn't restart because the 2nd server had possession of the lock. I shutdown the 2nd server and the lock was released, allowing me to restart them both.

Related questions

java.io.IOException: Broken pipe

We are currently migrating a legacy application to Jetty. And I have somehow an exception regarding a broken pipe. Java 6 Jetty 8.1.8 Spring 3.2.0 I am trying to migrate a Glassfish web application to Jetty. In our testing environment we are using …

Read More
Is there a way to pass jvm args via command line to maven?

Possible Duplicate: maven jetty plugin - how to control vm arguments? In particular, I want to do something like this: mvn -DjvmArgs="-Xmx2000m -Xms1000m -XX:PermSize=512m -XX:MaxPermSize=512m" jetty:run -Pmyprofile Oh, and I would like …

Read More
websocket closing connection automatically

I'm building an application in java that has an embedded websocket server based on jetty. The client is the default websocket implementation in google chrome. Everything is working ok, only if there is no transfer between server and client after …

Read More