mysql terminated with status 1?
This morning I noticed that my mysql server was not running. A look at the logs and I found the information below. While it is troubling that the mysqld service ran out of memory and was killed, it is more troubling that mysql could not restart.
Any ideas on why mysql could not respawn? How can I test to make sure that if the process is killed it will respawn?
Thank you.
387 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.475042] Out of memory: Kill process 810 (mysqld) score 232 or sacrifice child
388 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.475060] Killed process 810 (mysqld) total-vm:888108kB, anon-rss:139816kB, file-rss:0kB
389 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.655663] init: mysql main process (810) killed by KILL signal
390 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.655745] init: mysql main process ended, respawning
391 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.044805] type=1400 audit(1381408630.181:13): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=27754 comm="apparmor_parser"
392 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.676434] init: mysql main process (27763) terminated with status 1
393 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.676489] init: mysql main process ended, respawning
394 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.468923] init: mysql post-start process (27764) terminated with status 1
395 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.512363] type=1400 audit(1381408631.649:14): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=27800 comm="apparmor_parser"
396 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.681433] init: mysql main process (27804) terminated with status 1
397 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.681491] init: mysql respawning too fast, stopped
Answer
I would try running mysqld directly as a command, and looking at the output. It could be e.g. InnoDB corruption leading to stopping immediately after spawning, at which point upstart might try to respawn until apparmour stops it.