2 min read | by Jordi Prats
You can use kubectl drain to evict pods from a node and mark it as unschedulable to prevent new pods from get created on this node. It will allow the pod's containers to gracefully terminate, respecting the PodDisruptionBudgets with a few exceptions. Let's test it suing the following nodes:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
nauvoo.pet2cattle.com Ready control-plane,master 19d v1.20.4+k3s1
tycho.pet2cattle.com Ready <none> 26s v1.20.4+k3s1
If we try to use kubectl drain on one of the nodes we will get the following error:
$ kubectl drain tycho.pet2cattle.com
node/tycho.pet2cattle.com cordoned
error: unable to drain node "tycho.pet2cattle.com", aborting command...
There are pending nodes to be drained:
tycho.pet2cattle.com
error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/svclb-traefik-ndd2s
DaemonSet-managed pods cannot be deleted, to be able to safely drain a node containing pods that belong to a DaemonSet we will have to use the option --ignore-daemonsets:
$ kubectl drain tycho.pet2cattle.com --ignore-daemonsets
node/tycho.pet2cattle.com already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/svclb-traefik-ndd2s
node/tycho.pet2cattle.com drained
Checking again the list of nodes we will see that the node is marked as SchedulingDisabled: So no new pods are going to be scheduled on this node and since we have evicted all the relevant pods; we will be able to proceed safely with it's maintenance
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
nauvoo.pet2cattle.com Ready control-plane,master 19d v1.20.4+k3s1
tycho.pet2cattle.com Ready,SchedulingDisabled <none> 2m59s v1.20.4+k3s1
Once you are ready to put the node back into service, you can use kubectl uncordon to make the node schedulable again.
Posted on 14/04/2021