Failed to provision volume with StorageClass "gp3": invalid AWS VolumeType "gp3"

3 min read | by Jordi Prats

If we try to provision a gp3 Volume using a PVC on an EKS cluster we might get the invalid AWS VolumeType "gp3" error. This means we are using an outdated EBS CSI

We can check whether if we are able to use gp3 Volumes creating an StorageClass as follows:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  fsType: ext4 
volumeBindingMode: Immediate

And then creating a test volume using a PVC:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: testgp3
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp3

If we need to update the EBS CSI, the PVC will remain in Pending state and we will get the following error if we describe the PersistentVolumeClaim:

$ kubectl get pvc
NAME              STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pet2cattle-data   Bound     pvc-00b232c1-3603-4b15-80b0-a07d25bab876   20Gi       RWO            gp2            71m
testgp3           Pending                                                                        gp3            4s
$ kubectl describe pvc testgp3
Name:          testgp3
Namespace:     pet2cattle
StorageClass:  gp3
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  9s (x2 over 16s)  persistentvolume-controller  Failed to provision volume with StorageClass "gp3": invalid AWS VolumeType "gp3"

To install a newer version, we can follow AWS's instructions on how to install the latest EBS CSI. Once we have installed it we can create the gp2 and gp3 StorageClasses using the following yaml:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-gp2
provisioner: ebs.csi.aws.com
parameters:
  type: gp2
  fsType: ext4 
volumeBindingMode: WaitForFirstConsumer
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4 
volumeBindingMode: WaitForFirstConsumer

Once we have the new StorageClasses deployed we will have to update the default StorageClass. In a nutshell, we can do it using kubectl patch as follows:

kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass ebs-gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Repeating the kubectl get sc we can check that the new default StorageClass is the new gp2 StorageClass (ebs-gp2):

$ kubectl get sc
NAME                PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-gp2 (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   2d5h
ebs-gp3             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   2d5h
gp2                 kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   184d

We can, of course, set the gp3 instead, whatever suits best. If we create the PVC again we will see how the status transitions to bound:

$ kubectl get pvc
NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
testgp3                        Bound    pvc-00b232c1-3603-4b15-80b0-a07d25bab876   10Gi       RWO            gp3            7s

Posted on 21/09/2021

Categories