Where are Kubernetes' pods logfiles?

Question

When I run

$ kubectl logs <container>

I get the logs of my pods.

But where are the files for those logs?

Some sources says /var/log/containers/ others says /var/lib/docker/containers/ but I couldn't find my actual application's or pod's log.

Answer 1

Short Answer:

If you're using Docker, the stdout from each container are stored in /var/lib/docker/containers. But Kubernetes also creates a directory structure to help you find logs based on Pods, so you can find the container logs for each Pod running on a node at /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/.

---

Longer Answer:

Docker traps the stdout logs from each container and stores them in /var/lib/docker/containers on the host. If Kubernetes uses Docker as the container runtime, Docker will also store the containers logs in that location on the Kubernetes node. But since we don't run containers directly in Kubernetes (we run Pods), Kubernetes also creates the /var/log/pods/ and /var/log/containers directories to help us better organize the log files based on Pods.

Each directory within /var/log/pods/ stores the logs for a single Pod, and each are named using the structure <namespace>_<pod_name>_<pod_id>.

You can get the ID of a Pod by running kubectl get pod -n core gloo-76dffbd956-rmvdz -o jsonpath='{.metadata.uid}'. If you're used to using yq, you may find running kubectl get pod <pod_name> -o yaml | yq r - metadata.uid more straight-forward.

Within each /var/log/pods/<namespace>_<pod_name>_<pod_id>/ directory are more directories, each representing a container within the Pod. The name of these directories is equal to the name of the container. Lastly, when we look inside a /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/ directory, we'll find symbolic links to the log files stored by Docker inside /var/lib/docker/containers.

Similarly, inside the /var/log/containers/ directory are symlinks to a /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/ directory. These symlinks are named using the structure <pod_name>_<namespace>_<container_id>.