Setting an AWS CloudWatch Logs driver in docker
is done with log-driver=awslogs
and log-opt
, for example -
#!/bin/bash
docker run \
--log-driver=awslogs \
--log-opt awslogs-region=eu-central-1 \
--log-opt awslogs-group=whatever-group \
--log-opt awslogs-stream=whatever-stream \
--log-opt awslogs-create-group=true \
wernight/funbox \
fortune
I would like to use AWS CloudWatch logs in a Kubernetes cluster, where each pod contains a few Docker containers. Each deployment would have a separate Log Group, and each container would have a separate stream. I could not find a way to send the logging parameters to the docker containers via Kubernetes create
/ apply
.
How can I send the log-driver
and log-opt
parameters to a Docker container in a pod / deployment?
kubectl apply
kops
Kubernetes Logging Architecture
From what I understand, Kubernetes prefer Cluster-level logging to Docker logging driver.
We could use fluentd to collect, transform, and push container logs to CloudWatch Logs.
All you need is to create a fluentd DaemonSet with ConfigMap and Secret. Files can be found in Github. It has been tested with Kubernetes v1.7.5.
The following are some explains.
With DaemonSet, fluentd collect every container logs from the host folder /var/lib/docker/containers
.
fluent-plugin-kubernetes_metadata_filter plugin load the pod's metadata from Kubernetes API server.
The log record would be like this.
{
"log": "INFO: 2017/10/02 06:44:13.214543 Discovered remote MAC 62:a1:3d:f6:eb:65 at 62:a1:3d:f6:eb:65(kube-235)\n",
"stream": "stderr",
"docker": {
"container_id": "5b15e87886a7ca5f7ebc73a15aa9091c9c0f880ee2974515749e16710367462c"
},
"kubernetes": {
"container_name": "weave",
"namespace_name": "kube-system",
"pod_name": "weave-net-4n4kc",
"pod_id": "ac4bdfc1-9dc0-11e7-8b62-005056b549b6",
"labels": {
"controller-revision-hash": "2720543195",
"name": "weave-net",
"pod-template-generation": "1"
},
"host": "kube-234",
"master_url": "https://10.96.0.1:443/api"
}
}
Make some tags with Fluentd record_transformer filter plugin.
{
"log": "...",
"stream": "stderr",
"docker": {
...
},
"kubernetes": {
...
},
"pod_name": "weave-net-4n4kc",
"container_name": "weave"
}
fluent-plugin-cloudwatch-logs plugin send to AWS CloudWatch Logs.
With log_group_name_key
and log_stream_name_key
configuration, log group and stream name can be any field of the record.
<match kubernetes.**>
@type cloudwatch_logs
log_group_name_key pod_name
log_stream_name_key container_name
auto_create_stream true
put_log_events_retry_limit 20
</match>