How can I group labels in a Prometheus query?

checketts picture checketts · Jul 18, 2017 · Viewed 41.7k times · Source

If I have a metric with the following labels:

my_metric{group="group a"}  100
my_metric{group="group b"}  100
my_metric{group="group c"}  100
my_metric{group="misc group a"}  1
my_metric{group="misc group b"}  2
my_metric{group="misc group c"}  1
my_metric{group="misc group d"}  1

Is it possible to do a query or even a label_replace that combines the 'misc' groups together?

(I realize that the metric cardinality needs to be improved, and I've updated the app to fix it. However it left me with this question for if I wanted to fix the metrics via a query later)

Answer

tom.wilkie picture tom.wilkie · Jul 18, 2017

Yes, you can you use label replace to group all the misc together:

sum by (new_group) (
  label_replace(
    label_replace(my_metric, "new_group", "$1", "group", ".+"),
    "new_group", "misc", "group", "misc group.+"
  )
)

The inner label_replace copies all values from group into new_group, the outer overwrites those which match "misc group.+" with "misc", and we then sum by the "new_group" label. The reason for using a new label is the series would no longer be unique if we just overwrote the "group" label, and the sum wouldn't work.