Java app in Docker container does not log to syslog properly
My Goal
I have a couple of different containers running inside a host. They all share a volume with each other's /dev/log socket. The host forwards these logs to a central logging server. All the other containers logs are showing up on the host's /var/log/messages. The other containers are python programs which allows logging to directly append to the /dev/log unix domain socket.
Basic configuration
I have a docker container running a scala app from a jar file. The /dev/log sockets are shared as volumes between the host and container. The log4j.properties file for the app seems fine to me and it is set as follows:
# Root logger option
log4j.rootLogger=INFO, file, stdout, SYSLOG
# Direct log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=log/associationRules.log
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%c{1}:%L] %-5p <%X{jobID}> %m%n
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%c{1}:%L] %-5p <%X{jobID}> %m%n
# Log to syslog
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=localhost
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.conversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%c{1}:%L] %-5p <%X{jobID}> %m%n
log4j.appender.SYSLOG.Facility=LOCAL0
log4j.appender.SYSLOG.Threshold=debug
log4j.appender.SYSLOG.FacilityPrinting=true
The stdout and file logs are done properly, but SYSLOG does not seem to work correctly. The logs show up in the container's /var/log/syslog, but they do not show up on the host's /var/log/messages. I thought SyslogAppender would also output to /dev/log but it doesn't seem like it is doing that.
Additional details
Here is the output I get in the container's /var/log/syslog. I see some issues with the imuxsock not running, but I don't understand how the logs get stored on /var/log/syslog. The last two lines are from the app and are consistent with stdout and file appenders
Jul 6 18:07:18 26056b722779 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="25" x-info="http://www.rsyslog.com"] start
Jul 6 18:07:18 26056b722779 rsyslogd: cannot create '/dev/log': Address already in use
Jul 6 18:07:18 26056b722779 rsyslogd: imuxsock does not run because we could not aquire any socket
Jul 6 18:07:18 26056b722779 rsyslogd-3000: activation of module imuxsock failed
Jul 6 18:07:18 26056b722779 rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
Jul 6 18:07:18 26056b722779 rsyslogd-2145: activation of module imklog failed [try http://www.rsyslog.com/e/2145 ]
Jul 6 18:07:19 localhost local0: 2015-07-06 18:07:19 [AssocApp$:112] INFO <Undefined> Listening to queue ASSOCIATIONRULES
Jul 6 18:07:19 localhost local0: 2015-07-06 18:07:19 [AssocApp$:113] INFO <Undefined> Listening to queue ASSOCIATIONRULES
The container's base image is ubuntu 14.04 with java installed My CMD in the dockerfile for the container's image is:
CMD sudo service rsyslog start; java -cp /root/AssociationRules.jar AssocApp
and I start the container like this:
docker run -d -v /dev/log:/dev/log <IMAGE>
/etc/rsyslog.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
# Enable non-kernel facility klog messages
$KLogPermitNonKernelFacility on
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
#$PrivDropToUser syslog
#$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
$ModLoad imudp
$UDPServerRun 514
and /etc/rsyslog.d/50-default.conf
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info -/var/log/mail.info
#mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some "catch-all" log files.
#
#*.=debug;\
# auth,authpriv.none;\
# news.none;mail.none -/var/log/debug
#*.=info;*.=notice;*.=warn;\
# auth,authpriv.none;\
# cron,daemon.none;\
# mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
Answer
It seems to me you're overthinking this. If I read your question correctly, you want your Docker container to write it's syslog into the Host machine's syslog.
The error you've described probably arises because you're starting a second syslog daemon inside your container, which then wants to open the /dev/log socket (which is already there).
Several ideas come to mind (all untested, though. Enjoy with caution):
Why don't you configure Log4j do write into the remote syslog daemon directly? This would eliminate both the need for a syslog daemon inside the container, and the need to mount
/dev/loginto the container?# Log to syslog log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.syslogHost=<your-host-ip> # <-- INSERT HOST IP HERE log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayoutTo keep your container portable, best configure the IP address of the syslog server on container creation by using the
--add-hostflag:docker run -d --add-host sysloghost:<host-ip-here> <IMAGE>That way, you can simply use
sysloghostas a hostname in your Log4J configuration file.If you're insistent about running a syslog server inside the application container, you should be able (remember: all untested!) to configure it to relay all messages to a remote syslog server (somewhere in
/etc/rsyslog.confor/etc/rsyslog.d):*.* @sysloghost:512 # UDP forwarding # *.* @@sysloghost:512 # TCP forwardingBetter yet, why not run the syslog daemon inside it's own Docker container and link that container into your application containers?
Keep in mind that for both solutions, your syslog daemon on your host needs to be configured to listen on a TCP or UDP socket [ref]:
$ModLoad imudp
$UDPServerRun 514
Alternatively [ref]:
$ModLoad imtcp # needs to be done just once
$InputTCPMaxSessions 500
$InputTCPServerRun 514