Prometheus return no data when calculating a ratio of two metrics
I want to calculate a ratio of two metrics, but I get no data...
I have some metrics like:
fs_bytes{filesystem="/var",instance="localhost:9108",job="graphite",metric="Used"} 50.0
fs_bytes{filesystem="/var",instance="localhost:9108",job="graphite",metric="Total"} 100.0
When I try to do any operation (device, multiply, add, subtract) like:
fs_bytes{instance="localhost:9108",metric="Used"} / fs_bytes{instance="localhost:9108",metric="Total"}
Prometheus returned:
no data
When I query each metric alone in the Prometheus expression browser, I do get the metrics values.
What's wrong?
Answer
When prometheus is evaluating an expression, the operation implicitly apply to metric that share identical set of labels.
Despite the fact I specified the metric name and most labels, Prometheus was looking for metrics that have the same set of labels.
However, in this case, two metrics have different label values, so they can't match ! (one metric has metric="Used"the other has metric="Total". It could be that one of the metrics has some extra labels).
The solution is to use ignore (or on) to reduce the set of considered labels:
fs_bytes{instance="localhost:9108",metric="Used"} / ignoring(metric) fs_bytes{instance="localhost:9108",metric="Total"}
Read the fine manual ! (here)