bestby: Refresh Pods based on it's lifetime using a label on it's definition

Kubernetes Pod lifetime bestby

3 min read | by Jordi Prats

For Kubernetes a Deployment is an object to define a Pod that is intended to permanently run on the cluster, so there's no native way to refresh it's Pods. To do so we can install the bestby controller.

With this controller we can add a label with the time we want to stay runnning to the Pod (or a Pod template, such as Deployment or an StatefulSet). We can install it using helm:

helm repo add bestby https://pet2cattle.github.io/pod-best-by/
helm repo update
helm install -n bestby --create-namespace bestby bestby/bestby

To configure the amount of time we want the Pod to allow to run we can use any string that can be parsed using the ParseDuration go function:

ParseDuration parses a duration string. A duration string is a possibly signed sequence of 
decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" 
or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

If we are not setting any unit, it will default to minutes. So, if we want a Pod to expire after two minutes we'll have to set the following label:

labels:
  pod.kubernetes.io/lifetime: 2m

On a Deployment object it would look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-test
spec:
  replicas: 1
  selector:
    matchLabels:
      component: deploy-test
  template:
    metadata:
      labels:
        component: deploy-test
        pod.kubernetes.io/lifetime: 2m
    spec:
      containers:
      - name: demo-container
        image: "alpine:latest"
        command:
        - sleep
        - '24h'

If we deploy it with the bestby controller, we will be able to see how it will delete the Pod approximately every two minutes:

$ kubectl get pods --watch
NAME                           READY   STATUS              RESTARTS   AGE
deploy-test-65fbdd6cb7-fgt4l   0/1     ContainerCreating   0          7s
deploy-test-65fbdd6cb7-fgt4l   1/1     Running             0          13s
deploy-test-65fbdd6cb7-fgt4l   1/1     Terminating         0          3m
deploy-test-65fbdd6cb7-mvqfs   0/1     Pending             0          0s
deploy-test-65fbdd6cb7-mvqfs   0/1     Pending             0          0s
deploy-test-65fbdd6cb7-mvqfs   0/1     ContainerCreating   0          0s
deploy-test-65fbdd6cb7-mvqfs   1/1     Running             0          7s
deploy-test-65fbdd6cb7-fgt4l   0/1     Terminating         0          3m32s
deploy-test-65fbdd6cb7-fgt4l   0/1     Terminating         0          3m34s
deploy-test-65fbdd6cb7-fgt4l   0/1     Terminating         0          3m35s
deploy-test-65fbdd6cb7-mvqfs   1/1     Terminating         0          2m
deploy-test-65fbdd6cb7-x8fq4   0/1     Pending             0          0s
deploy-test-65fbdd6cb7-x8fq4   0/1     Pending             0          0s
deploy-test-65fbdd6cb7-x8fq4   0/1     ContainerCreating   0          0s
deploy-test-65fbdd6cb7-x8fq4   1/1     Running             0          10s
deploy-test-65fbdd6cb7-mvqfs   0/1     Terminating         0          2m34s
deploy-test-65fbdd6cb7-mvqfs   0/1     Terminating         0          2m34s
deploy-test-65fbdd6cb7-mvqfs   0/1     Terminating         0          2m35s
deploy-test-65fbdd6cb7-x8fq4   1/1     Terminating         0          2m
deploy-test-65fbdd6cb7-x8wzs   0/1     Pending             0          0s
deploy-test-65fbdd6cb7-x8wzs   0/1     Pending             0          0s
deploy-test-65fbdd6cb7-x8wzs   0/1     ContainerCreating   0          0s
deploy-test-65fbdd6cb7-x8wzs   1/1     Running             0          9s
deploy-test-65fbdd6cb7-x8fq4   0/1     Terminating         0          2m33s
deploy-test-65fbdd6cb7-x8fq4   0/1     Terminating         0          2m34s
deploy-test-65fbdd6cb7-x8fq4   0/1     Terminating         0          2m34s
deploy-test-65fbdd6cb7-x8wzs   1/1     Terminating         0          2m59s
deploy-test-65fbdd6cb7-b2rgt   0/1     Pending             0          1s
deploy-test-65fbdd6cb7-b2rgt   0/1     Pending             0          1s
deploy-test-65fbdd6cb7-b2rgt   0/1     ContainerCreating   0          2s
deploy-test-65fbdd6cb7-b2rgt   1/1     Running             0          10s
(...)

If we need a more precise termination of a container, we'll need to modify it to run the command together with the timeout command:

$ date; timeout 2m sleep 24h; date
Thu Mar  3 20:31:34 CET 2022
Thu Mar  3 20:33:34 CET 2022

Posted on 04/03/2022

Categories