How can I let the gitlab-ci-runner DinD image cache intermediate images?

Question

I have a Dockerfile that starts with installing the texlive-full package, which is huge and takes a long time. If I docker build it locally, the intermedate image created after installation is cached, and subsequent builds are fast.

However, if I push to my own GitLab install and the GitLab-CI build runner starts, this always seems to start from scratch, redownloading the FROM image, and doing the apt-get install again. This seems like a huge waste to me, so I'm trying to figure out how to get the GitLab DinD image to cache the intermediate images between builds, without luck so far.

I have tried using the --cache-dir and --docker-cache-dir for the gitlab-runner register command, to no avail.

Is this even something the gitlab-runner DinD image is supposed to be able to do?

My .gitlab-ci.yml:

build_job:
    script:
    - docker build --tag=example/foo .

My Dockerfile:

FROM php:5.6-fpm
MAINTAINER Roel Harbers <[email protected]>
RUN apt-get update && apt-get install -qq -y --fix-missing --no-install-recommends texlive-full
RUN echo Do other stuff that has to be done every build.

I use GitLab CE 8.4.0 and gitlab/gitlab-runner:latest as runner, started as

docker run -d --name gitlab-runner --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/local/gitlab-ci-runner/config:/etc/gitlab-runner \
    gitlab/gitlab-runner:latest \
; \

The runner is registered using:

docker exec -it gitlab-runner gitlab-runner register \
    --name foo.example.com \
    --url https://gitlab.example.com/ci \
    --cache-dir /cache/build/ \
    --executor docker \
    --docker-image gitlab/dind:latest \
    --docker-privileged \
    --docker-disable-cache false \
    --docker-cache-dir /cache/docker/ \
; \

This creates the following config.toml:

concurrent = 1
[[runners]]
    name = "foo.example.com"
    url = "https://gitlab.example.com/ci"
    token = "foobarsldkflkdsjfkldsj"
    tls-ca-file = ""
    executor = "docker"
    cache_dir = "/cache/build/"
    [runners.docker]
        image = "gitlab/dind:latest"
        privileged = true
        disable_cache = false
        volumes = ["/cache"]
        cache_dir = "/cache/docker/"

(I have experimented with different values for cache_dir, docker_cache_dir and disable_cache, all with the same result: no caching whatsoever)

Answer 1

I suppose there's no simple answer to your question. Before adding some details, I strongly suggest to read this blog article from the maintainer of DinD, which was originally named "do not use Docker in Docker for CI".

What you might try is declaring /var/lib/docker as a volume for your GitLab runner. But be warned, depending on your file-system drivers you may use AUFS in the container on an AUFS filesystem on your host, which is very likely to cause problems.

What I'd suggest to you is creating a separate Docker-VM, only for the runner(s), and bind-mount docker.sock from the VM into your runner-container. We are using this setup with GitLab with great success (>27.000 builds in about 12 months).

You can take a look at our runner with docker-compose support which is actually based on the shell-executor of GitLab's runner.