"no space left on device" when building with Jenkins and Docker

docker jenkins rancher
Matt picture Matt · Apr 24, 2019 · Viewed 8.3k times · Source

we use Jenkins to build Docker images and everything was find until last week when every time I try to build an image in Jenkins it fails with "Error response from daemon: Error processing tar file(exit status 1): write /app/node_modules/acorn/dist/acorn_loose.es.js: no space left on device" (the file it fails on can depend on the project). The image will build if I try using Docker in the server but fails on Jenkins.

I have tried removing old containers and images etc but to no avail. Disk space and inodes seem to be fine so I'm not sure what to try now. Any help is appreciated.

Result of "docker info":

Containers: 55
 Running: 48
 Paused: 0
 Stopped: 7
Images: 59
Server Version: 17.03.2-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.45-rancher
Operating System: RancherOS v1.1.0
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 19.61 GiB
Name: rancher
ID: Z7Z3:T3NW:N4O3:FKMZ:7KH6:FJ7R:TJ6A:FXLW:KNUL:WMRC:ED74:KHEM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 dockerhub.companysite.net:5000
 127.0.0.0/8
Live Restore Enabled: false

Result of "df -h":

Filesystem      Size  Used Avail Use% Mounted on
overlay          47G   27G   18G  60% /
tmpfs           9.9G     0  9.9G   0% /dev
tmpfs           9.9G     0  9.9G   0% /sys/fs/cgroup
/dev/sda1        47G   27G   18G  60% /.r
shm              64M     0   64M   0% /dev/shm
tmpfs           9.9G     0  9.9G   0% /sys/firmware

Result of "df -ih":

Filesystem     Inodes IUsed IFree IUse% Mounted on
overlay           13M  2.8M  9.8M   22% /
tmpfs            2.5M    16  2.5M    1% /dev
tmpfs            2.5M    15  2.5M    1% /sys/fs/cgroup
/dev/sda1         13M  2.8M  9.8M   22% /.r
shm              2.5M     1  2.5M    1% /dev/shm
tmpfs            2.5M     1  2.5M    1% /sys/firmware

Answer

Ardesco picture Ardesco · Apr 24, 2019

You may have a bunch of old dangling images left over that are causing problems, try:

docker volume rm $(docker volume ls -qf dangling=true)

(This does delete stuff, so you may want to check there is nothing you want to keep in any dangling volumes before doing this)

Related questions

Error "The input device is not a TTY"

I am running the following command from my Jenkinsfile. However, I get the error "The input device is not a TTY". docker run -v $PWD:/foobar -it cloudfoundry/cflinuxfs2 /foobar/script.sh Is there a way to run the script …

Read More
Docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

I am new to docker. I just tried to use docker in my local machine(Ubuntu 16.04) with Jenkins. I configured a new job with below pipeline script. node { stage('Build') { docker.image('maven:3.3.3').inside { sh 'mvn --version' } } } But it …

Read More
Rebuild Docker container on file changes

For running an ASP.NET Core application, I generated a dockerfile which build the application and copys the source code in the container, which is fetched by Git using Jenkins. So in my workspace, I do the following in the …

Read More