How to calculate cpu usage of pod in kube state metrics?
I monitor the eks cluster using both the kubernetes api and the kube state metrics remotely using prometheus.
in Kubernetes api we have a metrics container_cpu_usage_seconds_total which gives the cpu usage of the pod.
is there a similar metrics in kube-state-metrics which can give the cpu usage.
Actually I'm trying to get the cluster cpu usage which is totally different from kubernetes api and kube-state-metrics
following are the calculations.
kube-state-metrics:
sum(kube_pod_container_resource_requests_cpu_cores) / sum(kube_node_status_allocatable_cpu_cores) * 100 - This gives 101%
whereas the kube-state-metrics gives 12% which looks accurate to me.
kubernetes-api:
sum (rate (container_cpu_usage_seconds_total{id="/",kubernetes_io_hostname=~"^$Node$", job=~"$job$"}[5m])) / sum (machine_cpu_cores{kubernetes_io_hostname=~"^$Node$", job=~"$job$"}) * 100
I don't think there's any metric in kube-state-metric which gives cpu usage compared to kubernetes-api
Thanks in advance.
Answer
No, there is no (one) specific metric for cpu usage per container in kube_state_metric.
The value you got:
sum(kube_pod_container_resource_requests_cpu_cores) / sum(kube_node_status_allocatable_cpu_cores) * 100 = 101
may be wrong because metrics like kube_node_status_allocatable_cpu_cores and kube_pod_container_resource_requests_cpu_cores are marked as DEPRECATED.
At the same time, take notice that there is metric like kube_pod_container_resource_limits_cpu_cores.
Your container could have set resource limit, that's why probably your result exceed 100 %. If you have limit set per container check if resource limit is lower that resource request and then your calculation should looks like:
[sum(kube_pod_container_resource_requests_cpu_cores) - sum(kube_pod_container_resource_limits_cpu_cores)]/ sum(kube_node_status_allocatable_cpu_cores) * 100.
Take a look for every resource metrics in kube_state_metrics for container and nodes: node_metrics, pod_container_metrics.