Grafana HTTP Error Bad Gateway and Templating init failed errors

online picture online · Jan 19, 2018 · Viewed 17.2k times · Source

Use helm installed Prometheus and Grafana on minikube at local.

$ helm install stable/prometheus
$ helm install stable/grafana

Prometheus server, alertmanager grafana can run after set port-forward:

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9090

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9093

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=excited-crocodile-grafana,component=grafana" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 3000

enter image description here enter image description here

Add Data Source from grafana, got HTTP Error Bad Gateway error:

enter image description here

Import dashboard 315 from:

https://grafana.com/dashboards/315

Then check Kubernetes cluster monitoring (via Prometheus), got Templating init failed error:

enter image description here

Why?

Answer

svenwltr picture svenwltr · Jan 19, 2018

In the HTTP settings of Grafana you set Access to Proxy, which means that Grafana wants to access Prometheus. Since Kubernetes uses an overlay network, it is a different IP.

There are two ways of solving this:

  1. Set Access to Direct, so the browser directly connects to Prometheus.
  2. Use the Kubernetes-internal IP or domain name. I don't know about the Prometheus Helm-chart, but assuming there is a Service named prometheus, something like http://prometheus:9090 should work.