kubectl drain: prepare node for maintenance

2 min read

You can use kubectl drain to evict pods from a node and mark it as unschedulable to prevent new pods from get created on this node. It will allow the pod's containers to gracefully terminate, respecting the PodDisruptionBudgets with a few exceptions. Let's test it suing the following nodes:

$ kubectl get nodes
NAME                    STATUS   ROLES                  AGE   VERSION
nauvoo.pet2cattle.com   Ready    control-plane,master   19d   v1.20.4+k3s1
tycho.pet2cattle.com    Ready    <none>                 26s   v1.20.4+k3s1

If we try to use kubectl drain on one of the nodes we will get the following error:

$ kubectl drain tycho.pet2cattle.com
node/tycho.pet2cattle.com cordoned
error: unable to drain node "tycho.pet2cattle.com", aborting command...

There are pending nodes to be drained:
 tycho.pet2cattle.com
error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/svclb-traefik-ndd2s

DaemonSet-managed pods cannot be deleted, to be able to safely drain a node containing pods that belong to a DaemonSet we will have to use the option --ignore-daemonsets:

$ kubectl drain tycho.pet2cattle.com --ignore-daemonsets
node/tycho.pet2cattle.com already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/svclb-traefik-ndd2s
node/tycho.pet2cattle.com drained

Checking again the list of nodes we will see that the node is marked as SchedulingDisabled: So no new pods are going to be scheduled on this node and since we have evicted all the relevant pods; we will be able to proceed safely with it's maintenance

$ kubectl get nodes
NAME                    STATUS                     ROLES                  AGE     VERSION
nauvoo.pet2cattle.com   Ready                      control-plane,master   19d     v1.20.4+k3s1
tycho.pet2cattle.com    Ready,SchedulingDisabled   <none>                 2m59s   v1.20.4+k3s1

Once you are ready to put the node back into service, you can use kubectl uncordon to make the node schedulable again.


Posted on 14/4/2021

Categories