Kubernetes: cannot delete Pods with local storage

2 min read | by Jordi Prats

While draining a node it might fail with the message cannot delete Pods with local storage as follows:

$ kubectl drain tycho.pet2cattle.com --ignore-daemonsets
node/tycho.pet2cattle.com already cordoned
error: unable to drain node "tycho.pet2cattle.com", aborting command...

There are pending nodes to be drained:
 tycho.pet2cattle.com
error: cannot delete Pods with local storage (use --delete-emptydir-data to override): spinnaker-ampa/spin-rosco-658fdb4694-v99jt

This message tells us that there is a volume that uses local storage, for example using emptyDir:

apiVersion: v1
kind: Pod
metadata:
  name: shared-volume-pod
spec:
  containers:
  - image: busybox
    name: test-container
    volumeMounts:
    - mountPath: /data
      name: shared-volume
  volumes:
  - name: shared-volume
    emptyDir: {}

It's important to understand that any data that it is stored there will be lost while evicting this pod. This kind of volumes should only be used for temporary data and we should assume that it's data can be deleted at any time. Using the option --delete-emptydir-data we are telling kubectl drain that we understand this:

$ kubectl drain tycho.pet2cattle.com --ignore-daemonsets --delete-emptydir-data
node/tycho.pet2cattle.com already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-f5dcj
evicting pod spinnaker-ampa/spin-rosco-658fdb4694-v99jt
evicting pod spinnaker-ampa/spin-echo-86d57bfb8d-lfmwf
evicting pod spinnaker-ampa/spin-igor-98fbd9b54-lvs6d
evicting pod spinnaker-ampa/spin-gate-65686fb4b5-vg2dg
evicting pod spinnaker-ampa/spin-fiat-7ddfc67fdb-dqpkp
pod/spin-fiat-7ddfc67fdb-dqpkp evicted
pod/spin-gate-65686fb4b5-vg2dg evicted
pod/spin-echo-86d57bfb8d-lfmwf evicted
pod/spin-igor-98fbd9b54-lvs6d evicted
pod/spin-rosco-658fdb4694-v99jt evicted
node/tycho.pet2cattle.com evicted

Posted on 02/08/2021

Categories