Updating secrets using the External Secrets Operator

Kubernetes Secret ExternalSecret Operator update

3 min read | by Jordi Prats

When we are using an ExternalSecret the actual secret is stored elsewhere, being the External Secret Operator in charge of updating the Secret object with the datata if fetches from the SecretStore. What happens when the Secret is updated?

When the Secret is updated, depending on how we are using the secret we might need to restart the deployments using it:

Let's test it using the following ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: es-aws-demo
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: demo-ssm
    kind: SecretStore
  target:
    name: es-aws-demo
  data:
  - secretKey: demo
    remoteRef:
      key: demo

Once applied it's going to fetch the secret from the AWS Parameter Store and create the secret:

$ kubectl get externalsecret
NAME          STORE      REFRESH INTERVAL   STATUS         READY
es-aws-demo   demo-ssm   1m                 SecretSynced   True
$ kubectl get secret
NAME          TYPE     DATA   AGE
es-aws-demo   Opaque   1      15m

We can now deploy the following that is going to create a Pod with the Secret mounted as a file and an environment variable:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: test-eso
  labels:
    app: test-eso
spec:
  selector:
    matchLabels:
      app: test-eso
  replicas: 1
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: test-eso
    spec:
      containers:
      - name: test-eso
        image: alpine:latest
        imagePullPolicy: Always
        command:  ["sh", "-c", "while true; do echo ===; echo env; echo $ESO_ENV; echo; echo volume; cat /secret/demo; echo; sleep 1; done"]
        env:
        - name: ESO_ENV
          valueFrom:
            secretKeyRef:
              name: es-aws-demo
              key: demo
        volumeMounts:
          - name: secret
            mountPath: /secret
      volumes:
        - name: secret
          secret:
            secretName: es-aws-demo

Once we have the Pod running we can update the secret on the parameter store:

$ awstools ssm set demo newvalue --description "test ESO" --overwrite
6647ac56-494c-431a-a4f7-f3b60c68c4bf

As soon as the External Secret Operator fetches the data and updates the secret we'll be able to see on the Pod's log that it is able to read the new value from the file (but not from the environment variable)

$ kubectl logs test-eso-559f9fcd4c-b9cgl -f
(...)
===
env
value

volume
value
===
env
value

volume
newvalue

It won't be able to read the new value even if we create a new process:

$ kubectl exec -it test-eso-559f9fcd4c-b9cgl -- sh
/ # echo $ESO_ENV
value

To be able to use the new value when it is pushed using an environment variable we'll have to refresh the Pods.

Posted on 11/10/2022

Categories