MongoDB won't start after server crash

Question

My Ubuntu computer had crashed, and when I restarted it MongoDB wasn't working. I tried the following commands, and got the following output:

$ mongo
Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo.js:91
exception: connect failed

$ service mongodb status
mongodb stop/waiting

$ service mongodb restart
stop: Unknown instance: 
start: Rejected send message, 1 matched rules; type="method_call",
       sender=":1.57" (uid=1000 pid=2227 comm="start mongodb ")
       interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)"
       requested_reply="0"
       destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")

$ tail /var/log/mongodb/mongodb.log
[initandlisten] exception in initAndListen: 12596 old lock file, terminating
dbexit: 
[initandlisten] shutdown: going to close listening sockets...
[initandlisten] shutdown: going to flush diaglog...
[initandlisten] shutdown: going to close sockets...
[initandlisten] shutdown: waiting for fs preallocator...
[initandlisten] shutdown: closing all files...
[initandlisten] closeAllFiles() finished
dbexit: really exiting now

(Output reformatted to match website layout.)

What happened? How can I fix it?

Answer 1

The log file is telling you that you have an "old lock file". MongoDB keeps a lock file while it's running. It creates this file when it is started, and deletes it when it's stopped. When the computer crashes (or MongoDB crashes, e.g. via kill), this file is not deleted, and thus the database does not start. The existence of this file indicates unclean shutdown of MongoDB.

Two things can be done:

1. If this is a development machine and you haven't been using your database (and neither have your programs), you can remove the file manually. For MongoDB 2.2.2 running on Ubuntu 12.10, it's in /var/lib/mongodb/mongod.lock. For other versions, the file could be in a different path or it could be named mongo.lock.

2. The safer route is to follow MongoDB's Durability and Repair guide. In summary, for a machine with the above configuration, you should execute the following commands:

   sudo -u mongodb mongod --repair --dbpath /var/lib/mongodb/
   sudo service mongod start