mysql service fails to start/hangs up - timeout (Ubuntu, MariaDB)
I set up my first Ubuntu Server with Ubuntu 16.04, nginx, php7.0, MariaDB, nextcloud and external DynDNS using this tutorial here: Install Nextcloud 9 on Ubuntu 16.04
Everything worked fine but since I restarted the server the next day, nextcloud just shows me a blank page. After clicking through all logs of nginx, MariaDB and nextcloud, I found out that the mysql service just doesn't start. So run service mysql start and everything worked fine again (calling nextcloud from server as well as other workstations). I just wondered that the terminal didn't "close" the line. Like it was still working on the command. After about 5 minutes, the line "closes" and the following message appears:
"Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details."
Then the clients again just get a blank page in nextcloud. When I run the command and close the terminal immediately, clients get the access as well but still loses it after 5 minutes.
I tried backing up the nextcloud, sql and run apt-get purge --auto-remove mariadb-server. Then again run the MariaDB installation steps out of the tutorial with importing the backup sql instead of creating a new one. Didn't change everything.
Next try was update-rc.d mysql defaults and update-rc.d mysql enable. But after a restart just the blank page again. Access is only possible for 5 minutes by starting the service manual.
I also tried the BUM - BootUpManager but the service seems to be enabled. I saw you can start services out oft it manually as well. So tried it with mysql and surprise: nextcloud available for 5 minutes while BUM just hangs up.
I found mariadb.com/kb/en/mariadb/starting-and-stopping-mariadb-automatically/ as well but tried nothing of it because it seems like there is something else really wrong.
root@s1:~# systemctl status mariadb.service:
\u25cf mariadb.service - MariaDB database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset:
Drop-In: /etc/systemd/system/mariadb.service.d
\u2514\u2500migrated-from-my.cnf-settings.conf
Active: failed (Result: timeout) since Di 2016-12-06 14:52:51 CET; 55s ago
Process: 3565 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WS
Process: 3415 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR
Process: 3409 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START
Process: 3405 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/ru
Main PID: 3565 (code=exited, status=0/SUCCESS)
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] /usr/sbin
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] Event Sch
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 2147785536 [Note] InnoDB: F
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] InnoDB: S
Dez 06 14:52:49 s1 mysqld[3565]: 2016-12-06 14:52:49 3067387712 [Note] InnoDB: W
Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] InnoDB: S
Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] /usr/sbin
Dez 06 14:52:51 s1 systemd[1]: Failed to start MariaDB database server.
Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Unit entered failed state.
Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Failed with result 'timeout'.
root@s1:~# journalctl -xe:
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] Event Scheduler: Purging the queue. 0 events
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 2147785536 [Note] InnoDB: FTS optimize thread exiting.
Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] InnoDB: Starting shutdown...
Dez 06 14:52:49 s1 mysqld[3565]: 2016-12-06 14:52:49 3067387712 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer po
Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] InnoDB: Shutdown completed; log sequence number 111890806
Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] /usr/sbin/mysqld: Shutdown complete
Dez 06 14:52:50 s1 audit[3648]: AVC apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profi
Dez 06 14:52:50 s1 kernel: audit: type=1400 audit(1481032370.973:29): apparmor="DENIED" operation="sendmsg" info="Failed name lookup -
Dez 06 14:52:50 s1 audit[3565]: AVC apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profi
Dez 06 14:52:50 s1 kernel: audit: type=1400 audit(1481032370.973:30): apparmor="DENIED" operation="sendmsg" info="Failed name lookup -
Dez 06 14:52:51 s1 systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Unit entered failed state.
Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Failed with result 'timeout'.
Dez 06 14:54:54 s1 x11vnc[2665]: 06/12/2016 14:54:54 cursor_noshape_updates_clients: 1
Dez 06 14:55:16 s1 ntpd[1244]: 46.4.1.155 local addr 192.168.178.50 -> <null>
Dez 06 14:57:30 s1 ntpd[1244]: 89.238.66.98 local addr 192.168.178.50 -> <null>
Content in /ect/init.d (if useful):
#!/bin/bash
#
### BEGIN INIT INFO
# Provides: mysql
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Should-Start: $network $named $time
# Should-Stop: $network $named $time
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start and stop the mysql database server daemon
# Description: Controls the main MariaDB database server daemon "mysqld"
# and its wrapper script "mysqld_safe".
### END INIT INFO
#
set -e
set -u
${DEBIAN_SCRIPT_DEBUG:+ set -v -x}
test -x /usr/sbin/mysqld || exit 0
. /lib/lsb/init-functions
SELF=$(cd $(dirname $0); pwd -P)/$(basename $0)
CONF=/etc/mysql/my.cnf
MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"
# priority can be overriden and "-s" adds output to stderr
ERR_LOGGER="logger -p daemon.err -t /etc/init.d/mysql -i"
# Safeguard (relative paths, core dumps..)
cd /
umask 077
# mysqladmin likes to read /root/.my.cnf. This is usually not what I want
# as many admins e.g. only store a password without a username there and
# so break my scripts.
export HOME=/etc/mysql/
# Source default config file.
[ -r /etc/default/mariadb ] && . /etc/default/mariadb
## Fetch a particular option from mysql's invocation.
#
# Usage: void mysqld_get_param option
mysqld_get_param() {
/usr/sbin/mysqld --print-defaults \
| tr " " "\n" \
| grep -- "--$1" \
| tail -n 1 \
| cut -d= -f2
}
## Do some sanity checks before even trying to start mysqld.
sanity_checks() {
# check for config file
if [ ! -r /etc/mysql/my.cnf ]; then
log_warning_msg "$0: WARNING: /etc/mysql/my.cnf cannot be read. See README.Debian.gz"
echo "WARNING: /etc/mysql/my.cnf cannot be read. See README.Debian.gz" | $ERR_LOGGER
fi
# check for diskspace shortage
datadir=`mysqld_get_param datadir`
if LC_ALL=C BLOCKSIZE= df --portability $datadir/. | tail -n 1 | awk '{ exit ($4>4096) }'; then
log_failure_msg "$0: ERROR: The partition with $datadir is too full!"
echo "ERROR: The partition with $datadir is too full!" | $ERR_LOGGER
exit 1
fi
}
## Checks if there is a server running and if so if it is accessible.
#
# check_alive insists on a pingable server
# check_dead also fails if there is a lost mysqld in the process list
#
# Usage: boolean mysqld_status [check_alive|check_dead] [warn|nowarn]
mysqld_status () {
ping_output=`$MYADMIN ping 2>&1`; ping_alive=$(( ! $? ))
ps_alive=0
pidfile=`mysqld_get_param pid-file`
if [ -f "$pidfile" ] && ps `cat $pidfile` >/dev/null 2>&1; then ps_alive=1; fi
if [ "$1" = "check_alive" -a $ping_alive = 1 ] ||
[ "$1" = "check_dead" -a $ping_alive = 0 -a $ps_alive = 0 ]; then
return 0 # EXIT_SUCCESS
else
if [ "$2" = "warn" ]; then
echo -e "$ps_alive processes alive and '$MYADMIN ping' resulted in\n$ping_output\n" | $ERR_LOGGER -p daemon.debug
fi
return 1 # EXIT_FAILURE
fi
}
#
# main()
#
case "${1:-''}" in
'start')
sanity_checks;
# Start daemon
log_daemon_msg "Starting MariaDB database server" "mysqld"
if mysqld_status check_alive nowarn; then
log_progress_msg "already running"
log_end_msg 0
else
# Could be removed during boot
test -e /var/run/mysqld || install -m 755 -o mysql -g root -d /var/run/mysqld
# Start MariaDB!
/usr/bin/mysqld_safe "${@:2}" > /dev/null 2>&1 &
# 6s was reported in #352070 to be too little
for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-60}"); do
sleep 1
if mysqld_status check_alive nowarn ; then break; fi
log_progress_msg "."
done
if mysqld_status check_alive warn; then
log_end_msg 0
# Now start mysqlcheck or whatever the admin wants.
output=$(/etc/mysql/debian-start)
[ -n "$output" ] && log_action_msg "$output"
else
log_end_msg 1
log_failure_msg "Please take a look at the syslog"
fi
fi
;;
'stop')
# * As a passwordless mysqladmin (e.g. via ~/.my.cnf) must be possible
# at least for cron, we can rely on it here, too. (although we have
# to specify it explicit as e.g. sudo environments points to the normal
# users home and not /root)
log_daemon_msg "Stopping MariaDB database server" "mysqld"
if ! mysqld_status check_dead nowarn; then
set +e
shutdown_out=`$MYADMIN shutdown 2>&1`; r=$?
set -e
if [ "$r" -ne 0 ]; then
log_end_msg 1
[ "$VERBOSE" != "no" ] && log_failure_msg "Error: $shutdown_out"
log_daemon_msg "Killing MariaDB database server by signal" "mysqld"
killall -15 mysqld
server_down=
for i in `seq 1 600`; do
sleep 1
if mysqld_status check_dead nowarn; then server_down=1; break; fi
done
if test -z "$server_down"; then killall -9 mysqld; fi
fi
fi
if ! mysqld_status check_dead warn; then
log_end_msg 1
log_failure_msg "Please stop MariaDB manually and read /usr/share/doc/mariadb-server-10.1/README.Debian.gz!"
exit -1
else
log_end_msg 0
fi
;;
'restart')
set +e; $SELF stop; set -e
$SELF start
;;
'reload'|'force-reload')
log_daemon_msg "Reloading MariaDB database server" "mysqld"
$MYADMIN reload
log_end_msg 0
;;
'status')
if mysqld_status check_alive nowarn; then
log_action_msg "$($MYADMIN version)"
else
log_action_msg "MariaDB is stopped."
exit 3
fi
;;
'bootstrap')
# Bootstrap the cluster, start the first node
# that initiates the cluster
log_daemon_msg "Bootstrapping the cluster" "mysqld"
$SELF start "${@:2}" --wsrep-new-cluster
;;
*)
echo "Usage: $SELF start|stop|restart|reload|force-reload|status|bootstrap"
exit 1
;;
esac
Unfortunately, Google can't help me. I tried to explain as much as I can maybe this helps you in helping me. Thanks a lot!
Answer
In case you are bitten by this bug, the solution is given as a suggestion in the bug report:
echo "/usr/sbin/mysqld { }" > /etc/apparmor.d/usr.sbin.mysqldapparmor_parser -v -R /etc/apparmor.d/usr.sbin.mysqldsystemctl restart mariadb
Background
If you previously had MySQL installed, it activated an AppArmor profile which is incompatible with MariaDB. apt-get remove --purge only removes the profile, but does not deactivate/unload it. Only manually unloading it lets MariaDB work unhindered by AppArmor.