Generating range vectors from return values in Prometheus queries
I have a metric varnish_main_client_req of type counter and I want to set up an alert that triggers if the rate of requests drops/raises by a certain amount in a given time (e.g. "Amount of requests deviated in the last 2 min!").
Using the deriv() function should work much better than comparing relative values, but it can only be used with gauges. Is it possible to convert an ever increasing metric aka. counter to a rated metric aka. gauge?
Query: deriv(rate(varnish_main_client_req[2m])[5m])
Expectation: Prometheus calculates the rate of client requests over the last 2 mins and returns a derivative of the resulting values over the last 5 mins.
Actual result:
"error": "parse error at char 48: range specification must be preceded by a metric selector, but follows a *promql.Call instead"
Recording rules might be an option but it feels like a cheap workaround for something that should work with queries:
my_gauge_metric = rate(some_counter_metric[2m])
Answer
Yes, you need to use a recording rule for this.
Prometheus calculates the rate of client requests over the last 2 mins and returns a derivative of the resulting values over the last 5 mins.
Herein lies the problem - at what interval should Prometheus synthesise this data?