Docker daemon memory leak due to logs from long running process

memory docker lxc dockerfile
alpha_cod picture alpha_cod · Dec 23, 2014 · Viewed 18.7k times · Source

I have the following setup:

  1. Perl service running in a container and writing logs out to STDERR
  2. logspout to ship those logs out to a remote server for archiving

in a 600 MB RAM machine.

I also truncate the logs periodically at:

/var/lib/docker/containers/CID/CID-json.log

as suggested here to avoid 100% disk scenarios.

Problem

Docker daemon starts of with low memory usage, 1% initially and slowly increases to 40% after 2 days of running the container.

Reference

Docker daemon memory leak has been talked about in this issue and this issue. But both of them are closed now saying merged at a commit. Am running the latest major version of docker (Docker version 1.4.0, build 4595d4f), but still face a monotonically increasing memory usage issue.

EDIT: I did this experiment: Just run a bash process in the container, print out a lot of lines to STDERR, docker daemon process's memory usage accelerates very quickly

Does docker do some log buffering and doesn't release memory even if underlying log file (/var/lib/docker/containers/CID/CID-json.log) is cleared?

There's apparently no way to clear the logs. Will this commit solve this issue for long running tasks?

I don't know why docker daemon's memory usage keeps increasing. How do I debug this issue?

Answer

mhsmith picture mhsmith · Mar 6, 2015

There is still at least one outstanding issue relating to memory leaks with logs: https://github.com/docker/docker/issues/9139

Related questions

Docker error : no space left on device

I installed docker on a Debian 7 machine in the following way $ echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list $ sudo apt-get update $ curl -sSL https://get.docker.com/ubuntu/ | sudo sh …

Read More
Docker container and memory consumption

Assume I am starting a big number of docker containers which are based on the same docker image. It means that each docker container is running the same application. It could be the case that the application is big enough …

Read More
Java using much more memory than heap size (or size correctly Docker memory limit)

For my application, the memory used by the Java process is much more than the heap size. The system where the containers are running starts to have memory problem because the container is taking much more memory than the heap …

Read More