Prevent a Pod being evicted by the cluster autoscaler

2 min read | by Jordi Prats

The cluster autoscaler takes into consideration several factors when it chooses a node to remove (evicting it's Pods)

When choosing a node to remove it considers:

It also takes into account Pods that cannot be moved elsewhere due to any set of constraints like nodeSelectors, affinity or simply due to lack of resources

But we can also set an annotation to tell it not to evict a particular Pod:

$ kubectl describe pod ampa-5948679894-vcmnv -n ampa
Name:         ampa-5948679894-vcmnv
Namespace:    ampa
Priority:     0
Node:         scopuli.pet2cattle.com/10.12.16.41
Start Time:   Mon, 18 Oct 2021 19:56:53 +0200
Labels:       app.kubernetes.io/instance=ampa
              app.kubernetes.io/name=website
              pod-template-hash=5948679894
Annotations:  cluster-autoscaler.kubernetes.io/safe-to-evict: false

(...)

This also works the other way around, we can tell the cluster autoscaler that is is safe to evict a particular Pod that meets any of the other considerations using the same annotation set to true:

"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"

Posted on 28/10/2021

Categories