Liveness and readiness probe connection refused

Question

I keep getting this error when I try to setup liveness & readiness prob for my awx_web container

Liveness probe failed: Get http://POD_IP:8052/: dial tcp POD_IP:8052: connect: connection refused

Liveness & Readiness section in my deployment for the container awx_web

          ports:
          - name: http
            containerPort: 8052 # the port of the container awx_web
            protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: 8052
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /
              port: 8052
            initialDelaySeconds: 5
            periodSeconds: 5

if I test if the port 8052 is open or not from another pod in the same namespace as the pod that contains the container awx_web or if I test using a container deployed in the same pod as the container awx_web i get this (port is open)

/ # nc -vz POD_IP 8052
POD_IP  (POD_IP :8052) open

I get the same result (port 8052 is open) if I use netcat (nc) from the worker node where pod containing the container awx_web is deployed.

for info I use a NodePort service that redirect traffic to that container (awx_web)

type: NodePort
ports:
- name: http
  port: 80
  targetPort: 8052
  nodePort: 30100

Answer 1

I recreated your issue and it looks like your problem is caused by too small value of initialDelaySeconds for the liveness probe.

It takes more than 5s for awx container to open 8052 port. You need to wait a bit longer for it to start. I have found out that setting it to 15s is enough for me, but you may require some tweaking.