AWS EKS: How to resize Persistent Volumes

AWS EKS resize volume

3 min read | by Jordi Prats

On a AWS EKS cluster, at the time of this writing, by default you cannot resize volumes provisioned with the default gp2 StorageClass. This is because on the default StorageClass the allowVolumeExpansion is set to false, preventing the volume expansion:

$ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  78d

To be able to fix this, we can check that on the default definitions the nallowVolumeExpansion setting is not even present:

$ kubectl get sc gp2 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  (...)
  name: gp2
parameters:
  fsType: ext4
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

To be able to modify this StorageClass we can use kubectl patch as follows:

$ kubectl patch sc gp2 -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched

This setting will tell the Kubernetes CSI that the underlying volume (a EBS Volume) can be resized. Using kubectl get sc we can check how it changed ALLOWVOLUMEEXPANSION to true: 

$ kubectl get sc 
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   78d

We can see the current size of the volume using kubectl get pv or, by simply checking the volume size on the pod using df:

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS   REASON   AGE
pvc-a1448f38-5f28-492e-a09c-8a900b9fb43e   35Gi       RWO            Delete           Bound    pet2cattle/pet2cattle-static   gp2                     9d6h
$ kubectl exec -it pet2cattle-79979695b-7rmg6 -- df -hP
Filesystem      Size  Used Avail Use% Mounted on
overlay          20G   11G  9.5G  53% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/xvda1       20G   11G  9.5G  53% /tmp
shm              64M     0   64M   0% /dev/shm
/dev/xvdbg       35G  1.3G   34G   4% /opt/pet2cattle/static
tmpfs           3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           3.9G     0  3.9G   0% /proc/acpi
tmpfs           3.9G     0  3.9G   0% /proc/scsi
tmpfs           3.9G     0  3.9G   0% /sys/firmware

Bear in mind that identifying VolumeMounts with linux mounts on the container can be a complex task.

Finally, to resize the volume we can change the storage property to the new size we want to resize it to:

$ kubectl get pvc pet2cattle-data -o yaml | sed 's/storage: 35Gi/storage: 40Gi/g' | kubectl apply -f -
persistentvolumeclaim/pet2cattle-data configured

It can take a while for the CSI to make the changes but eventually we will be able to see how the volume have been resized:

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS   REASON   AGE
pvc-a1448f38-5f28-492e-a09c-8a900b9fb43e   40Gi       RWO            Delete           Bound    pet2cattle/pet2cattle-static   gp2                     9d6h
$ kubectl exec -it pet2cattle-79979695b-7rmg6 -- df -hP
Filesystem      Size  Used Avail Use% Mounted on
overlay          20G   11G  9.5G  53% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/xvda1       20G   11G  9.5G  53% /tmp
shm              64M     0   64M   0% /dev/shm
/dev/xvdbg       40G  1.3G   38G   4% /opt/pet2cattle/static
tmpfs           3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           3.9G     0  3.9G   0% /proc/acpi
tmpfs           3.9G     0  3.9G   0% /proc/scsi
tmpfs           3.9G     0  3.9G   0% /sys/firmware

Posted on 10/05/2021

Categories