CentOS 7 rsyslog DEBUG logs dropped for C/C++ modules
I am using rsyslog (rsyslog-7.4.7-7.el7_0.x86_64) on CentOS 7 (CentOS Linux release 7.1.1503 (Core)). We have some applications on it which is using syslog framework for logging. We have a lot of logs. At peak, it can be upto 50000 logs in one second. Our system was earlier running on CentOS 6.2 (and rsyslog 5.8) and we never observed any drop. After doing some search, we found that there is rate limiting. We are getting messages like "imjournal: begin to drop messages due to rate-limiting" in /var/log/messages and then "imjournal: 130886 messages lost due to rate-limiting". We tried different ways to disable or tune it without success. We tried the following.
1) Changes in /etc/rsyslog.conf
$ModLoad imjournal # provides access to the systemd journal
$imjournalRatelimitInterval 1
$imjournalRatelimitBurst 50000
Some other info from rsyslog.conf as follows. Didn't change anything here
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
We also saw that there is some rate limiting with imuxsock; but that we understand that that won't be used when OmitLocalLogging is ON
2) Changes in /etc/systemd/journald.conf
Storage=auto
RateLimitInterval=1s
RateLimitBurst=100000
Our application has modules in Java (using SLF4J and LOG4J) and modules in C/C++ (using syslog() call). For the C/C++ modules, we are missing DEBUG logs most of the time. But DEBUG logs of Java modules are apparently fine always.
Version of systemd is "systemd-208-20.el7.x86_64". The application and rsyslogd are on same machine.
Answer
With the latest update to systemd (219-19) on CentOS 7, the only way we were able to get our logging working again, without any rate limiting by journald or IMUXSock, was using the config changes below. This also decreased, but didn't completely eliminate the excessive CPU consumption by journald.
Add the following to /etc/rsyslog.conf after '$ModLoad imuxsock' and '$ModLoad imjournal':
$IMUXSockRateLimitInterval 0
$IMJournalRatelimitInterval 0
Set the following in /etc/systemd/journald.conf:
Storage=volatile
Compress=no
RateLimitInterval=0
MaxRetentionSec=5s
Restart journald and rsyslog to pickup the changes with:
systemctl restart systemd-journald.service
systemctl restart rsyslog.service
Prior to this last update to systemd, you could comment out '$ModLoad imjournal' in /etc/rsyslog.conf to resolve this, but that doesn't work any longer.