Dynamically provisioned PersistentVolumes using StatefulSet
6 min read | by Jordi Prats
The basic idea behind a StatefulSet is to be able to manage stateful workloads on Kubernetes, unlike Deployments, creating a unique identity for each Pod using a common spec.
With this in mind we might just copy the Pod's template from a Deployment to a StatefulSet object to make it stateful, but it's not always quite that simple.
You'll be able to find all the object's definitions on the pet2cattle/kubernetes-statefullset-vs-deployment repository on GitHub.
Given the following Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-test
spec:
replicas: 1
selector:
matchLabels:
component: deploy-test
template:
metadata:
labels:
component: deploy-test
spec:
volumes:
- name: empty-dir
emptyDir: {}
containers:
- name: file-generator
image: "alpine:latest"
command:
- sleep
- '24h'
volumeMounts:
- mountPath: /test
name: empty-dir
Let's blindly copy it's spec.template into a StatefulSet:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: sts-test
spec:
serviceName: default
replicas: 1
selector:
matchLabels:
component: sts-test
volumeClaimTemplates:
- metadata:
name: sts-test
labels:
component: sts-test
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
template:
metadata:
labels:
component: sts-test
spec:
volumes:
- name: empty-dir
emptyDir: {}
containers:
- name: file-generator
image: "alpine:latest"
command:
- sleep
- '24h'
volumeMounts:
- mountPath: /test
name: empty-dir
If we deploy both objects, we will be able to see how the Deployment creates a Pod with hash on it's name. Meanwhile, the StatefulSet will give it a friendlier name:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deploy-test-57bb4d58bf-c67ck 1/1 Running 0 79s
sts-test-0 1/1 Running 0 37s
The fact it is using a friendlier name doesn't really mean anything: What's important it to note is that a Deployment, Pods are meant to be interchanged (cattle). Using a StatefulSet, on the other hand, each Pod has it's own identity (pets).
Another important difference, we will be able to see how for the StatefulSet will create a PersistentVolume for the emptyDir volume:
$ kubectl get pv | grep sts
pvc-30131c25-2c6c-4883-9f30-58793c72b442 10Gi RWO Delete Bound test/sts-test-sts-test-0 ebs-gp2 68s
Does this means that the data on the Volume is persistent? Actually, it's not.
Let's try to write some data on the volume using the Pod created by the Deployment and then delete the Pod:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deploy-test-57bb4d58bf-c67ck 1/1 Running 0 2m14s
$ kubectl exec -it deploy-test-57bb4d58bf-c67ck -- ls -l /test
total 0
$ kubectl exec -it deploy-test-57bb4d58bf-c67ck -- touch /test/persistence
$ kubectl exec -it deploy-test-57bb4d58bf-c67ck -- ls -l /test
total 0
-rw-r--r-- 1 root root 0 Feb 16 22:31 persistence
$ kubectl delete pod deploy-test-57bb4d58bf-c67ck
pod "deploy-test-57bb4d58bf-c67ck" deleted
As expected, if we check the volume again it will be empty:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deploy-test-57bb4d58bf-878x2 1/1 Running 0 57s
sts-test-0 1/1 Running 0 4m35s
$ kubectl exec -it deploy-test-57bb4d58bf-878x2 -- ls -l /test
total 0
How is this going to be handled on a StatefulSet? The volume that will be used if going to be a PersistentVolume but it's data will be wiped at each container restart. We can repeat the same test on the Pod created by the StatefulSet:
$ kubectl exec -it sts-test-0 -- ls -l /test
total 0
$ kubectl exec -it sts-test-0 -- touch /test/persistence
$ kubectl exec -it sts-test-0 -- ls -l /test
total 0
-rw-r--r-- 1 root root 0 Feb 16 22:34 persistence
$ kubectl delete pod sts-test-0
pod "sts-test-0" deleted
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deploy-test-57bb4d58bf-878x2 1/1 Running 0 3m27s
sts-test-0 1/1 Running 0 17s
$ kubectl exec -it sts-test-0 -- ls -l /test
total 0
It's not emptying the data because we are deleting the Pod, even with a rollout restart we'll get the same result:
$ kubectl exec -it sts-test-0 -- ls -l /test
total 0
$ kubectl exec -it sts-test-0 -- touch /test/persistence
$ kubectl exec -it sts-test-0 -- ls -l /test
total 0
-rw-r--r-- 1 root root 0 Feb 16 22:39 persistence
$ kubectl rollout restart sts sts-test
statefulset.apps/sts-test restarted
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sts-test-0 1/1 Terminating 0 4m33s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sts-test-0 1/1 Running 0 34s
$ kubectl exec -it sts-test-0 -- ls -l /test
total 0
The volume is wiped because we are using an emptyDir that will make sure that every time a Pod is deleted/restarted/... the data in the emptyDir is deleted permanently:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: sts-test
spec:
(...)
template:
metadata:
labels:
component: sts-test
spec:
volumes:
- name: empty-dir
emptyDir: {}
containers:
- name: file-generator
image: "alpine:latest"
command:
- sleep
- '24h'
volumeMounts:
- mountPath: /test
name: empty-dir
If we want data to be persistent across restarts, what we really want is a dynamically provisioned PersistentVolume (one for each replica). To accomplish this we can use the volumeClaimTemplates. We only need to make sure it's name matches the name of the volume we are mounting using volumeMounts (no need to declare the volume under spec.template.spec.volumes)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: sts-vt-test
spec:
serviceName: default
replicas: 1
selector:
matchLabels:
component: sts-vt-test
volumeClaimTemplates:
- metadata:
name: sts-vt-test
labels:
component: sts-vt-test
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
template:
metadata:
labels:
component: sts-vt-test
spec:
containers:
- name: file-generator
image: "alpine:latest"
command:
- sleep
- '24h'
volumeMounts:
- mountPath: /test
name: sts-vt-test
Once this new StatefulSet object is deployed, we can create a test file and then delete the Pod as we previously did, but this time the file will be there:
$ kubectl apply -f sts-volume-template.yaml
statefulset.apps/sts-vt-test created
$ kubectl get pv | grep sts
pvc-1bde765e-cd2b-4de9-a1f5-92095dccc11a 10Gi RWO Delete Bound test/sts-vt-test-sts-vt-test-0 ebs-gp2 12s
$ kubectl exec -it sts-vt-test-0 -- ls -l /test
total 16
drwx------ 2 root root 16384 Feb 16 23:54 lost+found
$ kubectl exec -it sts-vt-test-0 -- touch /test/persistence
$ kubectl exec -it sts-vt-test-0 -- ls -l /test
total 16
drwx------ 2 root root 16384 Feb 16 23:54 lost+found
-rw-r--r-- 1 root root 0 Feb 16 23:55 persistence
$ kubectl delete pod sts-vt-test-0
pod "sts-vt-test-0" deleted
$ kubectl exec -it sts-vt-test-0 -- ls -l /test
total 16
drwx------ 2 root root 16384 Feb 16 23:54 lost+found
-rw-r--r-- 1 root root 0 Feb 16 23:55 persistence
Posted on 21/02/2022