Kubernetes HPA -- Unable to get metrics for resource memory: no metrics returned from resource metrics API

Question

Good afternoon

I'm just starting with Kubernetes, and I'm working with HPA (HorizontalPodAutoscaler):

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: find-complementary-account-info-1
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: find-complementary-account-info-1
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageUtilization: 50

But when I try to see the metrics on the console, it doesn't show me:

    [dockermd@tmp108 APP-MM-ConsultaCuentaPagadoraPospago]$ kubectl get hpa
    NAME                                REFERENCE                                      TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
    find-complementary-account-info-1   Deployment/find-complementary-account-info-1   <unknown>/50%   2         5         2          143m

My manifiest is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: find-complementary-account-info-1
  labels:
    app: find-complementary-account-info-1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: find-complementary-account-info-1
  template:
    metadata:
      labels:
        app: find-complementary-account-info-1
    spec:
      containers:
      - name: find-complementary-account-info-1
        image: find-complementary-account-info-1:latest
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: "350Mi"
          requests:
            memory: "300Mi"
        ports:
        - containerPort: 8083
        env:
        - name: URL_CONNECTION_BD
          value: jdbc:oracle:thin:@10.161.6.15:1531/DEFAULTSRV.WORLD
        - name: USERNAME_CONNECTION_BD
          valueFrom:
            secretKeyRef:
              name: credentials-bd-pers
              key: user_pers
        - name: PASSWORD_CONNECTION_BD
          valueFrom:
            secretKeyRef:
              name: credentials-bd-pers
              key: password_pers
apiVersion: v1
kind: Service
metadata:
  name: svc-find-complementary-account-info-1
  labels:
    app: find-complementary-account-info-1
  namespace: default
spec:
  selector:
    app: find-complementary-account-info-1
  type: LoadBalancer
  ports:
   -
    protocol: TCP
    port: 8083
    targetPort: 8083
    nodePort: 30025
  externalIPs:
    - 10.161.174.68

Show me the tag

Also if I perform a kubect describe hpa, it throws me the following:

[dockermd@tmp108 APP-MM-ConsultaCuentaPagadoraPospago]$ kubectl describe hpa find-complementary-account-info-1
Name:                                                     find-complementary-account-info-1
Namespace:                                                default
Labels:                                                   <none>
Annotations:                                              <none>
CreationTimestamp:                                        Thu, 29 Oct 2020 13:57:58 -0400
Reference:                                                Deployment/find-complementary-account-info-1
Metrics:                                                  ( current / target )
  resource memory on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                             2
Max replicas:                                             5
Deployment pods:                                          2 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource memory: no metrics returned from resource metrics API
Events:
  Type     Reason                   Age                     From                       Message
  ----     ------                   ----                    ----                       -------
  Warning  FailedGetResourceMetric  4m49s (x551 over 144m)  horizontal-pod-autoscaler  unable to get metrics for resource memory: no metrics returned from resource metrics API

I am not working in the cloud, I am working in an on-premise environment configured with Bare-Metal

Also install metrics-server:

[dockermd@tmp108 APP-MM-ConsultaCuentaPagadoraPospago]$ kubectl get deployment metrics-server -n kube-system
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           177m

What will I lack? Could someone give me a hand?

[dockermd@tmp108 certs]$ kubectl describe pod metrics-server -n kube-system
Name:         metrics-server-5f4b6b9889-6pbv8
Namespace:    kube-system
Priority:     0
Node:         tmp224/10.164.21.169
Start Time:   Thu, 29 Oct 2020 13:27:19 -0400
Labels:       k8s-app=metrics-server
              pod-template-hash=5f4b6b9889
Annotations:  cni.projectcalico.org/podIP: 10.244.119.140/32
              cni.projectcalico.org/podIPs: 10.244.119.140/32
Status:       Running
IP:           10.244.119.140
IPs:
  IP:           10.244.119.140
Controlled By:  ReplicaSet/metrics-server-5f4b6b9889
Containers:
  metrics-server:
    Container ID:  docker://f71d26dc2c8e787ae9551faad66f9588a950bf0a6d0d5cb90ff11ceb219e9b37
    Image:         k8s.gcr.io/metrics-server-amd64:v0.3.6
    Image ID:      docker-pullable://k8s.gcr.io/metrics-server-amd64@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b
    Port:          4443/TCP
    Host Port:     0/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
    State:          Running
      Started:      Thu, 29 Oct 2020 13:27:51 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-4mn92 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  metrics-server-token-4mn92:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-4mn92
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/arch=amd64
                 kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

Answer 1

The error unable to get metrics for resource memory: no metrics returned from resource metrics API is not caused by faulty HPA but by metrics-server that is not able to scrape any metrics.

Most of the time this error is caused by missing commands in metrics-server deployment. It can be done by adding following arguments in the deployment.

  --kubelet-insecure-tls
  --kubelet-preferred-address-types=InternalIP,ExternalIP