No space on device with Jenkins and Docker - how to cleanup properly

Question

We're running Jenkins (version 2.60.1) on an Ubuntu 16.04.1 server. One of the issues we've been running into recently is that we routinely get the error "no space left on device".

I understand when using Docker there needs to be a strict clean-up process due to the files that are left behind and taking up unnecessary space.

We're using the CloudBees Docker Build and Publish plugin to handle the build and push to AWS ECS. I thought about removing all the unused Images. The thing is if I login to the Jenkins instance (over SSH) and try to run the docker command it gives - "Cannot connect to the Docker daemon. Is the docker daemon running on this host?"

I suppose somehow I need to do this from within Jenkins environment or part of the plugin?

Anyone dealt with this before or have some advice? - I'd really appreciate it.

Answer 1

Docker < 1.13

For Docker older than 1.13 you can do following for cleaning up some space on your device:

docker ps -a | grep -i 'exited' | awk '{print $1}' | xargs docker rm > /dev/null 2>&1 &
docker images -a | grep "<none>" | awk '{print $3}' | xargs docker rmi > /dev/null 2>&1 &

Alternatively you can try running following docker command:

docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

It will clean old orphan containers and will remove images tagged with <none>. I use these two formulas on one of my CI servers and it works fine. Before that I was facing similar to your issue (no space left on device).

Cleaning orphan volumes

docker volume rm $(docker volume ls -qf dangling=true)
docker volume ls -qf dangling=true | xargs -r docker volume rm

Docker >= 1.13

Docker 1.13 introduces docker system prune command (https://docs.docker.com/engine/reference/commandline/system_prune/). Alternatively you can run:

• docker image prune
• docker volume prune
• docker container prune

You can run those commands as a part of your Jenkins pipeline. In one of the projects I work on we run cleanup after building new Docker images during the release process. Try it as well to fix "Cannot connect to the Docker daemon. Is the docker daemon running on this host?" problem.