Finding the layers and layer sizes for each Docker image

Question

For research purposes I'm trying to crawl the public Docker registry ( https://registry.hub.docker.com/ ) and find out 1) how many layers an average image has and 2) the sizes of these layers to get an idea of the distribution.

However I studied the API and public libraries as well as the details on the github but I cant find any method to:

• retrieve all the public repositories/images (even if those are thousands I still need a starting list to iterate through)
• find all the layers of an image
• find the size for a layer (so not an image but for the individual layer).

Can anyone help me find a way to retrieve this information?

Thank you!

EDIT: is anyone able to verify that searching for '*' in Docker registry is returning all the repositories and not just anything that mentions '*' anywhere? https://registry.hub.docker.com/search?q=*

Answer 1

You can find the layers of the images in the folder /var/lib/docker/aufs/layers; provide if you configured for storage-driver as aufs (default option)

Example:

 docker ps -a
 CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
 0ca502fa6aae        ubuntu              "/bin/bash"         44 minutes ago      Exited (0) 44 seconds ago                       DockerTest

Now to view the layers of the containers that were created with the image "Ubuntu"; go to /var/lib/docker/aufs/layers directory and cat the file starts with the container ID (here it is 0ca502fa6aae*)

 root@viswesn-vm2:/var/lib/docker/aufs/layers# cat    0ca502fa6aaefc89f690736609b54b2f0fdebfe8452902ca383020e3b0d266f9-init 
 d2a0ecffe6fa4ef3de9646a75cc629bbd9da7eead7f767cb810f9808d6b3ecb6
 29460ac934423a55802fcad24856827050697b4a9f33550bd93c82762fb6db8f
 b670fb0c7ecd3d2c401fbfd1fa4d7a872fbada0a4b8c2516d0be18911c6b25d6
 83e4dde6b9cfddf46b75a07ec8d65ad87a748b98cf27de7d5b3298c1f3455ae4

This will show the result of same by running

root@viswesn-vm2:/var/lib/docker/aufs/layers# docker history ubuntu
IMAGE               CREATED             CREATED BY                                         SIZE                COMMENT
d2a0ecffe6fa        13 days ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B                 
29460ac93442        13 days ago         /bin/sh -c sed -i 's/^#\s*\   (deb.*universe\)$/   1.895 kB            
b670fb0c7ecd        13 days ago         /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   194.5 kB            
83e4dde6b9cf        13 days ago         /bin/sh -c #(nop) ADD file:c8f078961a543cdefa   188.2 MB 

To view the full layer ID; run with --no-trunc option as part of history command.

docker history --no-trunc ubuntu