Huge files in Docker containers

docker dockerfile docker-container
Eleanore picture Eleanore · Sep 15, 2016 · Viewed 11.1k times · Source

I need to create a Docker image (and consequently containers from that image) that use large files (containing genomic data, thus reaching ~10GB in size).

How am I supposed to optimize their usage? Am I supposed to include them in the container (such as COPY large_folder large_folder_in_container)? Is there a better way of referencing such files? The point is that it sounds strange to me to push such container (which would be >10GB) in my private repository. I wonder if there is a way of attaching a sort of volume to the container, without packing all those GBs together.

Thank you.

Answer

dnephin picture dnephin · Sep 15, 2016

Is there a better way of referencing such files?

If you already have some way to distribute the data I would use a "bind mount" to attach a volume to the containers.

docker run -v /path/to/data/on/host:/path/to/data/in/container <image> ...

That way you can change the image and you won't have to re-download the large data set each time.

If you wanted to use the registry to distribute the large data set, but want to manage changes to the data set separately, you could use a data volume container with a Dockerfile like this:

FROM tianon/true
COPY dataset /dataset
VOLUME /dataset

From your application container you can attach that volume using:

docker run -d --name dataset <data volume image name>
docker run --volumes-from dataset <image> ...

Either way, I think https://docs.docker.com/engine/tutorials/dockervolumes/ are what you want.

Related questions

Couldn't connect to Docker daemon at http+docker://localhost with docker-compose

I have this React app that I want to run on an Apache HTTP Docker container. So I created a Dockerfile that works with sudo docker build and sudo docker run <name> FROM httpd:2.4 COPY ./dist/ /usr/local/…

Read More
Does a RHEL7 docker container need subscription?

I have a rhel7 host with docker in it. I have build a image with also rhel7 as base image. But when I run the container & do yum update or try to install in package with yum, it gives …

Read More
How to pass local machine's SSH key to docker container?

I'm trying to build a docker image from Dockerfile and one of the steps that need to be taken is installing a dependency that is only available via private Gitlab repository. This means the container will need to have access …

Read More