How to fix ingress 404 default backend

Question

I started to use Kubernetes few month ago and I actually migrate my microServices to my rancher cluster (RKE). Everything works good, my deployments are good and services too. I would like use ingress.

Everything looks good, services are find by ingress and pods are find by services. However when I try to go to the website, I have a 404 error page from ingress controller.

You can see my configuration for juste two paths : one nginx and on grafana. Someone knows how can i fix it and use ingress to do my reverse proxy ?

Thank you so much for your help.

I trie to use rewrite-target without result and add-base-url is deprecated.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    field.cattle.io/creatorId: user-cg5r7
    field.cattle.io/ingressState: '{"bXktaW5ncmVzcy9kZWZhdWx0L3d3dy5zY29sLWVhLm92aC8vbmdpbngvNDI=":""}'
    field.cattle.io/publicEndpoints: '[{"addresses":["51.68.226.21"],"port":80,"protocol":"HTTP","serviceName":"default:nginx-services","ingressName":"default:my-ingress","hostname":"www.scol-ea.ovh","path":"/nginx","allNodes":true}]'
  creationTimestamp: "2019-08-31T10:54:25Z"
  generation: 2
  labels:
    cattle.io/creator: or antoine
  name: my-ingress
  namespace: default
  resourceVersion: "106239"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/my-ingress
  uid: b27b7b20-cbdd-11e9-b16b-fa163ea73397
spec:
  rules:
  - host: www.scol-ea.ovh
    http:
      paths:
      - backend:
          serviceName: nginx-sample
          servicePort: 80
        path: /nginx
      - backend:
          serviceName: prometheus-grafana
          servicePort: http
        path : /grafana

------------------------
apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/targetWorkloadIds: '["deployment:default:nginx-sample"]'
    workload.cattle.io/targetWorkloadIdNoop: "true"
    workload.cattle.io/workloadPortBased: "true"
  creationTimestamp: "2019-08-31T10:03:47Z"
  labels:
    cattle.io/creator: norman
  name: nginx-sample
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1beta2
    controller: true
    kind: deployment
    name: nginx-sample
    uid: 57af9603-cb2a-11e9-b16b-fa163ea73397
  resourceVersion: "102071"
  selfLink: /api/v1/namespaces/default/services/nginx-sample
  uid: 9fffe98c-cbd6-11e9-b16b-fa163ea73397
spec:
  clusterIP: 10.43.183.187
  ports:
  - name: 80tcp02
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    workload.user.cattle.io/workloadselector: deployment-default-nginx-sample
  sessionAffinity: None
  type: ClusterIP

----------------------------
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-08-30T13:44:00Z"
  labels:
    app: prometheus-grafana
    chart: grafana-0.0.31
    heritage: Tiller
    io.cattle.field/appId: prometheus
    release: prometheus
  name: prometheus-grafana
  namespace: default
  resourceVersion: "2536"
  selfLink: /api/v1/namespaces/default/services/prometheus-grafana
  uid: 38ebd878-cb2c-11e9-b16b-fa163ea73397
spec:
  clusterIP: 10.43.142.143
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app: prometheus-grafana
  sessionAffinity: None
  type: ClusterIP

Answer 1

If you are using nginx as deployment proxy in the kubernetes and facing the issue then the below configuration can resolve the redirection issue.

Either you can include the below variables in the grafan.ini file or can expose as env variables as shown below,

GF_SERVER_DOMAIN=abc.google.com
GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:/grafana

So the Grafana deployment will look like the below one.

        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: grafana
          namespace: monitoring
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: grafana
            spec:
              containers:
              - name: grafana
                image: grafana/grafana:latest
                env:
                - name: GF_SERVER_DOMAIN
                  value: "abc.google.com"
                - name: GF_SERVER_ROOT_URL
                  value: "%(protocol)s://%(domain)s:/grafana"

                ports:
                - name: grafana
                  containerPort: 3000
                resources:
                  limits:
                    memory: "2Gi"
                    cpu: "1000m"
                  requests: 
                    memory: "1Gi"
                    cpu: "500m"
                volumeMounts:
                  - mountPath: /var/lib/grafana
                    name: grafana-storage-volume
                  - mountPath: /etc/grafana/provisioning/datasources
                    name: grafana-datasources
                    readOnly: false

And you have to update the nginx server location block as below,

       location /grafana {
           proxy_pass http://grafana.monitoring.svc.cluster.local:3000;
           rewrite ^/grafana/(.*) /$1 break;
           proxy_set_header Host $host;
        }