How to calculate containers' cpu usage in kubernetes with prometheus as monitoring?

kubernetes prometheus
Haoyuan Ge picture Haoyuan Ge · Oct 30, 2016 · Viewed 58.5k times · Source

I want to calculate the cpu usage of all pods in a kubernetes cluster. I found two metrics in prometheus may be useful:

container_cpu_usage_seconds_total: Cumulative cpu time consumed per cpu in seconds.
process_cpu_seconds_total: Total user and system CPU time spent in seconds.

Cpu Usage of all pods = increment per second of sum(container_cpu_usage_seconds_total{id="/"})/increment per second of sum(process_cpu_seconds_total)

However, I found every second's increment of container_cpu_usage{id="/"} larger than the increment of sum(process_cpu_seconds_total). So the usage may be larger than 1...

Answer

Camil picture Camil · Nov 3, 2016

This I'm using to get CPU usage at cluster level:

sum (rate (container_cpu_usage_seconds_total{id="/"}[1m])) / sum (machine_cpu_cores) * 100

I also track the CPU usage for each pod.

sum (rate (container_cpu_usage_seconds_total{image!=""}[1m])) by (pod_name)

I have a complete kubernetes-prometheus solution on GitHub, maybe can help you with more metrics: https://github.com/camilb/prometheus-kubernetes

enter image description here

enter image description here

Related questions

How to get number of pods running in prometheus

I am scraping the kubernetes metrics from prometheus and would need to extract the number of running pods. I can see container_last_seen metrics but how should i get no of pods running. Can someone help on this?

Read More
How to monitor disk usage of kubernetes persistent volumes?

I have container_fs_usage_bytes with prometheus to monitor container root fs, but it seems that there is no metrics for other volumes in cAdvisor.

Read More
Grafana HTTP Error Bad Gateway and Templating init failed errors

Use helm installed Prometheus and Grafana on minikube at local. $ helm install stable/prometheus $ helm install stable/grafana Prometheus server, alertmanager grafana can run after set port-forward: $ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" …

Read More