How do I access this Kubernetes service via kubectl proxy?

aknuds1 picture aknuds1 · Jul 18, 2017 · Viewed 23.4k times · Source

I want to access my Grafana Kubernetes service via the kubectl proxy server, but for some reason it won't work even though I can make it work for other services. Given the below service definition, why is it not available on http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana?

grafana-service.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: monitoring
  name: grafana
  labels:
    app: grafana
spec:
  type: NodePort
  ports:
  - name: web
    port: 3000
    protocol: TCP
    nodePort: 30902
  selector:
    app: grafana

grafana-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: monitoring
  name: grafana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:4.1.1
        env:
        - name: GF_AUTH_BASIC_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: user
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: password
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/grafana-storage
        ports:
        - name: web
          containerPort: 3000
        resources:
          requests:
            memory: 100Mi
            cpu: 100m
          limits:
            memory: 200Mi
            cpu: 200m
      - name: grafana-watcher
        image: quay.io/coreos/grafana-watcher:v0.0.5
        args:
          - '--watch-dir=/var/grafana-dashboards'
          - '--grafana-url=http://localhost:3000'
        env:
        - name: GRAFANA_USER
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: user
        - name: GRAFANA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: password
        resources:
          requests:
            memory: "16Mi"
            cpu: "50m"
          limits:
            memory: "32Mi"
            cpu: "100m"
        volumeMounts:
        - name: grafana-dashboards
          mountPath: /var/grafana-dashboards
      volumes:
      - name: grafana-storage
        emptyDir: {}
      - name: grafana-dashboards
        configMap:
          name: grafana-dashboards

The error I'm seeing when accessing the above URL is "no endpoints available for service "grafana"", error code 503.

Answer

Stefan R picture Stefan R · Jul 19, 2017

As Michael says, quite possibly your labels or namespaces are mismatching. However in addition to that, keep in mind that even when you fix the endpoint, the url you're after (http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana) might not work correctly.

Depending on your root_url and/or static_root_path grafana configuration settings, when trying to login you might get grafana trying to POST to http://localhost:8001/login and get a 404.

Try using kubectl port-forward instead:

kubectl -n monitoring port-forward [grafana-pod-name] 3000

then access grafana via http://localhost:3000/

https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/