Kubernetes deployment status in Jenkins

Question

I am writing a Jenkins Global pipeline library where I have a stage to deploy my docker image to K8s cluster. So after building my docker image during CI process I am promoting(deploying) the image to multiple environments(sequentially lower to higher). So, to get the correct status of deployment after running

kubectl apply -f Application-k8s-file.yaml

I used following command in a shell step.

kubectl rollout status deployment deployment_name

Things goes well if my deployment does not have error but if my deployment has some error(might be some code bug, application does not start) then this command kubectl rollout status deployment <deployment name> runs infinitely(as k8s retries again and again to redeploy) and my Jenkins job runs infinitely(till the Job timeout).

So to find a hack I tried a logic to put the timeout on this command and calculations are something like this:

timeout = (number of pods * liveness probe time + number of pods* 10) seconds

Not sure if this calculation is correct or not.

My code snippet looks like this

        sh(returnStdout: true,script:"#!/bin/sh +e\n timeout --preserve-status ${timeout_value} kubectl rollout status deployment ${deploymentName} --kubeconfig='/opt/kubernetes-secrets/${env}/kubeconfig' 2>tmpfile; echo \$? > tmpfile1")
    def readThisFile = readFile "tmpfile1.txt"

def var=readThisFile.toInteger()

           if(var==0)
           {
             echo "deployment successful"
           }
       else{"do something else"}

This works well initially but later I find that k8s "kubectl rollout status deployment " command doesn't give exit code 0 until all the pods get scheduled and old get terminated completely which sometimes take time.

What I basically want is a minimal calculated timeout value.

My K8s file have parameters like this:

   spec:
     minReadySeconds: 30

    livenessProbe:
      httpGet:
        path: /ping
        port: 80
      initialDelaySeconds: 45
      periodSeconds: 5
      timeoutSeconds: 60
    name: test-dummyservice
    ports:
    - containerPort: 80
    readinessProbe:
      httpGet:
        path: /health
        port: 80
      initialDelaySeconds: 60
      periodSeconds: 120
      timeoutSeconds: 60

I did not find anything specific related to this in K8s documentation. Anyone facing same challenge?

Answer 1

You should take a look at progressDeadlineSeconds. Once this exceeds the deadline the rollout status will exit out.

kubectl rollout status deployment ng                                                                                                                      
Waiting for rollout to finish: 2 out of 7 new replicas have been updated...
error: deployment "ng" exceeded its progress deadline

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#progress-deadline-seconds