Docker Fluentd Logging Driver For multiline

logging docker fluentd
cheng wee picture cheng wee · Sep 21, 2015 · Viewed 10.8k times · Source

I am trying to create a centralized logging system using fluentd for a docker environment. Currently, i able to send the docker log to fluentd using fluentd docker logging driver which is a much cleaner solution compare to reading the docker log file using in_tail method. However, i am currently facing the issue on multi lines log issue.

enter image description here

As you can see from the picture above, the multi lines log are out of order which is very confusing for user. Is there any way this can be solved?

Thanks.

Cw

Answer

Abhishek Galoda picture Abhishek Galoda · Sep 19, 2017

Using fluent-plugin-concat pluging helped me in fixing above problem.

Adding these lines in fluent-conf 

 <filter **>
  @type concat
  key log
  stream_identity_key container_id
  multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}
  multiline_end_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}
</filter>

Where my regular expression is checking for DateTimeStamp in Logs where each line starts with and Date and Timestamp (pay attention to "log":"2017-09-21 15:03:27.289) below

2017-09-21T15:03:27Z    tag     {"container_id":"11b0d89723b9c812be65233adbc51a71507bee04e494134258b7af13f089087f","container_name":"/bel_osc.1.bc1k2z6lke1d7djeq5s28xjyl","source":"stdout","log":"2017-09-21 15:03:27.289  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.6"}
2017-09-21T15:03:28Z    tag     {"container_id":"11b0d89723b9c812be65233adbc51a71507bee04e494134258b7af13f089087f","container_name":"/bel_osc.1.bc1k2z6lke1d7djeq5s28xjyl","source":"stdout","log":"2017-09-21 15:03:28.191  INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext"}

Also, I had to add below lines in Dockerfile to install the plugin

RUN ["gem", "install", "fluent-plugin-concat", "--version", "2.1.0"] 
#Works with Fluentd v0.14-debian

Though this regular expression doesn't work well when an exception occurs, but still much better than before. Fluentd Link, for reference.

Related questions

Where is the Docker daemon log?

Where is the Docker daemon log? Oddly cannot find an answer to this via man, StackOverflow or Docker Docs. Note I am not asking for the docker container STDOUT, but the daemon log for troubleshooting communications between the client and …

Read More
How to redirect docker container logs to a single file?

I want to redirect all the logs of my docker container to single log file to analyse them. I tried docker logs container > /tmp/stdout.log 2>/tmp/stderr.log but this gives log in two different file. I …

Read More
Redirecting command output in docker

I want to do some simple logging for my server which is a small Flask app running in a Docker container. Here is the Dockerfile # Dockerfile FROM dreen/flask MAINTAINER dreen WORKDIR /srv # Get source RUN mkdir -p /srv COPY …

Read More