Helm: How to wait for the deployment before assuming it has succeeded
4 min read | by Jordi Prats
When you upgrade your application using helm what it really does is to just apply the deployments and other objects upgrades. If your new deployment fails to start for some reason (such as missing image) you won't notice it until you actually check the kubernetes cluster.
Helm install and upgrade commands include a couple of options to assist in checking the deployments: --wait and --timeout
- --wait: It tells helm to wait for the deplyment to succeed before setting as successful
- -timeout: We can also set a timeout, by default is 300 seconds (5 minutes)
Let's test it out, first we can take a look at the current helm history:
$ helm history pet2cattle
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
84 Sat Jan 9 12:04:35 2021 superseded pet2cattle-2 3.22 Upgrade complete
85 Sun Jan 10 13:10:58 2021 superseded pet2cattle-2 3.23 Upgrade complete
86 Sun Jan 10 13:14:59 2021 superseded pet2cattle-2 3.24 Upgrade complete
87 Fri Jan 15 21:25:06 2021 superseded pet2cattle-2 3.25 Upgrade complete
88 Sat Jan 30 12:51:53 2021 superseded pet2cattle-2 3.27 Upgrade complete
89 Sun Jan 31 14:18:37 2021 superseded pet2cattle-2 3.28 Upgrade complete
90 Sun Jan 31 17:00:43 2021 superseded pet2cattle-2.1 3.28 Upgrade complete
91 Sun Jan 31 17:05:41 2021 superseded pet2cattle-2.1 3.29 Upgrade complete
92 Sun Jan 31 18:37:20 2021 superseded pet2cattle-2.1 3.31 Upgrade complete
93 Sun Jan 31 18:47:19 2021 deployed pet2cattle-2.1 3.32 Upgrade complete
To test a failing upgrade I have modified the image tag to a non existing so it will fail:
$ helm upgrade pet2cattle -f pet2cattle_values.yaml --wait .
Error: UPGRADE FAILED: timed out waiting for the condition
After timing out, on the helm history we will see that the deploy is marked as failed:
$ helm history pet2cattle
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
85 Sun Jan 10 13:10:58 2021 superseded pet2cattle-2 3.23 Upgrade complete
86 Sun Jan 10 13:14:59 2021 superseded pet2cattle-2 3.24 Upgrade complete
87 Fri Jan 15 21:25:06 2021 superseded pet2cattle-2 3.25 Upgrade complete
88 Sat Jan 30 12:51:53 2021 superseded pet2cattle-2 3.27 Upgrade complete
89 Sun Jan 31 14:18:37 2021 superseded pet2cattle-2 3.28 Upgrade complete
90 Sun Jan 31 17:00:43 2021 superseded pet2cattle-2.1 3.28 Upgrade complete
91 Sun Jan 31 17:05:41 2021 superseded pet2cattle-2.1 3.29 Upgrade complete
92 Sun Jan 31 18:37:20 2021 superseded pet2cattle-2.1 3.31 Upgrade complete
93 Sun Jan 31 18:47:19 2021 deployed pet2cattle-2.1 3.32 Upgrade complete
94 Tue Feb 2 19:43:06 2021 failed pet2cattle-2.1 3.33 Upgrade "pet2cattle" failed: timed out waiting for the condition
Nevertheless, deployments and their pods are going to be still in place:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pet2cattle-s3sync-d8488f8c-9q8j9 1/1 Running 0 2d
pet2cattle-845645df85-k2s8k 1/1 Running 0 43h
pet2cattle-s3sync-d7948c5f8-zp9w9 0/1 ImagePullBackOff 0 10m
pet2cattle-7bf686db6d-qmmt9 0/1 ImagePullBackOff 0 10m
So we will have to rollback the upgrade using helm rollback to a working release:
$ helm rollback pet2cattle 93
Rollback was a success! Happy Helming!
Once we have rolled back the upgrade we won't have the failed pods on the kubernetes cluster:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pet2cattle-s3sync-d8488f8c-9q8j9 1/1 Running 0 2d
pet2cattle-845645df85-k2s8k 1/1 Running 0 43h
pet2cattle-s3sync-d7948c5f8-zp9w9 0/1 Terminating 0 11m
Back on the helm history we will be able to see that the app version 3.33 failed to deploy and we had to rollbacl to version 3.32:
$ helm history pet2cattle
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
85 Sun Jan 10 13:10:58 2021 superseded pet2cattle-2 3.23 Upgrade complete
86 Sun Jan 10 13:14:59 2021 superseded pet2cattle-2 3.24 Upgrade complete
87 Fri Jan 15 21:25:06 2021 superseded pet2cattle-2 3.25 Upgrade complete
88 Sat Jan 30 12:51:53 2021 superseded pet2cattle-2 3.27 Upgrade complete
89 Sun Jan 31 14:18:37 2021 superseded pet2cattle-2 3.28 Upgrade complete
90 Sun Jan 31 17:00:43 2021 superseded pet2cattle-2.1 3.28 Upgrade complete
91 Sun Jan 31 17:05:41 2021 superseded pet2cattle-2.1 3.29 Upgrade complete
92 Sun Jan 31 18:37:20 2021 superseded pet2cattle-2.1 3.31 Upgrade complete
93 Sun Jan 31 18:47:19 2021 superseded pet2cattle-2.1 3.32 Upgrade complete
94 Tue Feb 2 19:43:06 2021 failed pet2cattle-2.1 3.33 Upgrade "pet2cattle" failed: timed out waiting for the condition
95 Tue Feb 2 19:54:10 2021 deployed pet2cattle-2.1 3.32 Rollback to 93
Posted on 24/02/2021