Helm: How to wait for the deployment before assuming it has succeeded

When you upgrade your application using helm what it really does is to just apply the deployments and other objects upgrades. If your new deployment fails to start for some reason (such as missing image) you won't notice it until you actually check the kubernetes cluster.

Helm install and upgrade commands include a couple of options to assist in checking the deployments: --wait and --timeout

  • --wait: It tells helm to wait for the deplyment to succeed before setting as successful
  • -timeout: We can also set a timeout, by default is 300 seconds (5 minutes)

Let's test it out, first we can take a look at the current helm history:

jprats@shuvak:~/git/helm-pet2cattle$ helm history pet2cattle
REVISION  UPDATED                   STATUS      CHART           APP VERSION DESCRIPTION     
84        Sat Jan  9 12:04:35 2021  superseded  pet2cattle-2    3.22        Upgrade complete
85        Sun Jan 10 13:10:58 2021  superseded  pet2cattle-2    3.23        Upgrade complete
86        Sun Jan 10 13:14:59 2021  superseded  pet2cattle-2    3.24        Upgrade complete
87        Fri Jan 15 21:25:06 2021  superseded  pet2cattle-2    3.25        Upgrade complete
88        Sat Jan 30 12:51:53 2021  superseded  pet2cattle-2    3.27        Upgrade complete
89        Sun Jan 31 14:18:37 2021  superseded  pet2cattle-2    3.28        Upgrade complete
90        Sun Jan 31 17:00:43 2021  superseded  pet2cattle-2.1  3.28        Upgrade complete
91        Sun Jan 31 17:05:41 2021  superseded  pet2cattle-2.1  3.29        Upgrade complete
92        Sun Jan 31 18:37:20 2021  superseded  pet2cattle-2.1  3.31        Upgrade complete
93        Sun Jan 31 18:47:19 2021  deployed    pet2cattle-2.1  3.32        Upgrade complete

To test a failing upgrade I have modified the image tag to a non existing so it will fail:

$ helm upgrade pet2cattle -f pet2cattle_values.yaml --wait .
Error: UPGRADE FAILED: timed out waiting for the condition

After timing out, on the helm history we will see that the deploy is marked as failed:

$ helm history pet2cattle
REVISION  UPDATED                   STATUS      CHART           APP VERSION DESCRIPTION                                                     
85        Sun Jan 10 13:10:58 2021  superseded  pet2cattle-2    3.23        Upgrade complete                                                
86        Sun Jan 10 13:14:59 2021  superseded  pet2cattle-2    3.24        Upgrade complete                                                
87        Fri Jan 15 21:25:06 2021  superseded  pet2cattle-2    3.25        Upgrade complete                                                
88        Sat Jan 30 12:51:53 2021  superseded  pet2cattle-2    3.27        Upgrade complete                                                
89        Sun Jan 31 14:18:37 2021  superseded  pet2cattle-2    3.28        Upgrade complete                                                
90        Sun Jan 31 17:00:43 2021  superseded  pet2cattle-2.1  3.28        Upgrade complete                                                
91        Sun Jan 31 17:05:41 2021  superseded  pet2cattle-2.1  3.29        Upgrade complete                                                
92        Sun Jan 31 18:37:20 2021  superseded  pet2cattle-2.1  3.31        Upgrade complete                                                
93        Sun Jan 31 18:47:19 2021  deployed    pet2cattle-2.1  3.32        Upgrade complete                                                
94        Tue Feb  2 19:43:06 2021  failed      pet2cattle-2.1  3.33        Upgrade "pet2cattle" failed: timed out waiting for the condition

Nevertheless, deployments and their pods are going to be still in place:

$ kubectl get pods
NAME                                     READY   STATUS             RESTARTS   AGE
pet2cattle-s3sync-d8488f8c-9q8j9         1/1     Running            0          2d
pet2cattle-845645df85-k2s8k              1/1     Running            0          43h
pet2cattle-s3sync-d7948c5f8-zp9w9        0/1     ImagePullBackOff   0          10m
pet2cattle-7bf686db6d-qmmt9              0/1     ImagePullBackOff   0          10m

So we will have to rollback the upgrade using helm rollback to a working release:

$ helm rollback pet2cattle 93
Rollback was a success! Happy Helming!

Once we have rolled back the upgrade we won't have the failed pods on the kubernetes cluster:

$ kubectl get pods
NAME                                     READY   STATUS        RESTARTS   AGE
pet2cattle-s3sync-d8488f8c-9q8j9         1/1     Running       0          2d
pet2cattle-845645df85-k2s8k              1/1     Running       0          43h
pet2cattle-s3sync-d7948c5f8-zp9w9        0/1     Terminating   0          11m

Back on the helm history we will be able to see that the app version 3.33 failed to deploy and we had to rollbacl to version 3.32:

$ helm history pet2cattle
REVISION  UPDATED                   STATUS      CHART           APP VERSION DESCRIPTION                                                     
85        Sun Jan 10 13:10:58 2021  superseded  pet2cattle-2    3.23        Upgrade complete                                                
86        Sun Jan 10 13:14:59 2021  superseded  pet2cattle-2    3.24        Upgrade complete                                                
87        Fri Jan 15 21:25:06 2021  superseded  pet2cattle-2    3.25        Upgrade complete                                                
88        Sat Jan 30 12:51:53 2021  superseded  pet2cattle-2    3.27        Upgrade complete                                                
89        Sun Jan 31 14:18:37 2021  superseded  pet2cattle-2    3.28        Upgrade complete                                                
90        Sun Jan 31 17:00:43 2021  superseded  pet2cattle-2.1  3.28        Upgrade complete                                                
91        Sun Jan 31 17:05:41 2021  superseded  pet2cattle-2.1  3.29        Upgrade complete                                                
92        Sun Jan 31 18:37:20 2021  superseded  pet2cattle-2.1  3.31        Upgrade complete                                                
93        Sun Jan 31 18:47:19 2021  superseded  pet2cattle-2.1  3.32        Upgrade complete                                                
94        Tue Feb  2 19:43:06 2021  failed      pet2cattle-2.1  3.33        Upgrade "pet2cattle" failed: timed out waiting for the condition
95        Tue Feb  2 19:54:10 2021  deployed    pet2cattle-2.1  3.32        Rollback to 93     

Posted on 24/02/2021

Categories