Using alert annotations in an alertmanager receiver
I've got an alert configured like this:
ALERT InstanceDown
IF up == 0
FOR 30s
ANNOTATIONS {
summary = "Server {{ $labels.Server }} is down.",
description = "{{ $labels.Server }} ({{ $labels.job }}) is down for more than 30 seconds."
}
The slack receiver looks like this:
receivers:
- name: general_receiver
slack_configs:
- api_url: https://hooks.slack.com/services/token
channel: "#telemetry"
title: "My summary"
text: "My description"
Is it possible to use the annotations in my receiver? This github comment indicates it is but I haven't been able to get anything to work from it.
Answer
You need to define your own template (I just hat to walk that path). For a brief example see: https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/
All alerts here have at least a summary and a runbook annotation. runbook contains an Wiki URL. I've defined the following template (as described in the blog article above) to include them into the slack message body:
{{ define "__slack_text" }}
{{ range .Alerts }}{{ .Annotations.summary }}
{{ if gt (len .Labels) (len .GroupLabels) }}({{ with .Labels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}
{{ end }}<{{ (index .Alerts 0).GeneratorURL }}|Source> | {{ if .CommonAnnotations.runbook }}<{{ .CommonAnnotations.runbook }}|:notebook_with_decorative_cover: Runbook>{{ else }}<https://wiki.some.where/Runbooks|:exclamation:*NO RUNBOOK*:exclamation:>{{ end }}
{{ end }}
{{ define "slack.default.text" }}{{ template "__slack_text" . }}{{ end }}
The templates overwrites the slack.default.text so there is no need to reference it in the receiver configuration.
Defaults can be found in source as well as the "documentation":
- https://github.com/prometheus/alertmanager/blob/v0.4.2/template/default.tmpl
- https://github.com/prometheus/alertmanager/blob/v0.4.2/template/template.go
For basics and details about the golang templating language:
For completeness' sake, the eventual template:
{{ define "__slack_text" }}
{{ range .Alerts }}{{ .Annotations.text }}{{ end }}
{{ end }}
{{ define "__slack_title" }}
{{ range .Alerts }}{{ .Annotations.title }}{{ end }}
{{ end }}
{{ define "slack.default.text" }}{{ template "__slack_text" . }}{{ end }}
{{ define "slack.default.title" }}{{ template "__slack_title" . }}{{ end }}