Using AWS Secrets Manager secrets and Systems Manager parameters in Kubernetes with KES

4 min read | by Jordi Prats

By using the Kubernetes External Secrets we can use external secret management systems, like AWS Secrets Manager or Vault, to securely add secrets in Kubernetes.

This is achieved by by using the ExternalSecret object which declares how to fetch the secret data, while the KES controller converts the ExternalSecrets to Secrets. The conversion is completely transparent to Pods that can access Secrets normally.

KES deprecation

Due to a go rewrite, KES has been deprecated since the release of the External Secrets Operator, with it we'll be able to retrieve Secrets from multiple backends easily. Checkout how it can be retrieve secrets from Vault

Install and setup Kubernetes External Secrets

To install KES on an AWS EKS using the AWS Secrets Manager or the Systems Manager (parameter store) we will need to create an IRSA role and attach a policy to be able to fetch the secrets.

For the parameter store we just need to allow ssm:GetParameter to the parameter (it can also be a wildcard) like so:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:GetParameter",
            "Resource": [
                "arn:aws:ssm:eu-west-1:123456789123:parameter/example/*"
            ]
        }
    ]
}

For the AWS Secrets Manager we will need to be able to use the KMS keys and at the same time be able to fetch the secret, so the policy would look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey",
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:eu-west-1:123456789123:key/deadbeef-ffffff-dead-beef-abcdef1234cc",
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "arn:aws:secretsmanager:eu-west-1:123456789123:secret:example/password-K5RGAp"
            ]
        }
    ]
}

We can merge both policies into one like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:GetParameter",
            "Resource": [
                "arn:aws:ssm:eu-west-1:123456789123:parameter/example/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey",
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:eu-west-1:123456789123:key/deadbeef-ffffff-dead-beef-abcdef1234cc",
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "arn:aws:secretsmanager:eu-west-1:123456789123:secret:example/password-K5RGAp"
            ]
        },
    ]
}

On the KES hel chart we'll need to set the ServiceAccount annotation to be able to use this policy:

serviceAccount:
  create: true
  annotations:
    "eks.amazonaws.com/role-arn": "arn:..."

AWS Systems Manager

Once the controller is ready, to create an ExternalSecret fetching a secret from the parameter store we will have to create the following example specifying the backend and the key fo fetch like follows. The backendType needs to be systemManager and the key, unlike with AWS Sercrets Manager starts with a slash. The name is going to be the key we are going to use to retrieve the secret:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: pet2cattle-ssm-keys
spec:
  backendType: systemManager
  data:
    - key: /example-ssm
      name: name_secret

AWS Sercrets Manager

On the other hand, if we are using AWS Sercrets Manager, the backendType will be secretsManager and the key doesn't need to start with a slash:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: pet2cattle-sm-keys
spec:
  backendType: secretsManager
  data:
    - key: example-ssm
      name: name_secret

Create ExternalSecret

Once deployed we can check it's status using kubectl get externalsecret:

$ kubectl get externalsecret
NAME                  LAST SYNC   STATUS    AGE
pet2cattle-sm-keys   7s          SUCCESS   49s

It's describe will help us debug any issue it might arise:

$ kubectl describe externalsecret
Name:         pet2cattle-sm-keys
Namespace:    pet2cattle
Labels:       <none>
Annotations:  <none>
API Version:  kubernetes-client.io/v1
Kind:         ExternalSecret
Metadata:
(...)
Spec:
  Backend Type:  secretsManager
  Data:
    Key:   pet2cattle-gitlab-sm-key
    Name:  name_secret
Status:
  Last Sync:            2021-10-02T19:36:19.313Z
  Observed Generation:  1
  Status:               SUCCESS
Events:                 <none>

We can check the actual secret as usual:

$ kubectl get secret
NAME                               TYPE                                  DATA   AGE
default-token-r92gc                kubernetes.io/service-account-token   3      155s
sh.helm.release.v1.pet2cattle.v1   helm.sh/release.v1                    1      16s
pet2cattle-sm-keys                 Opaque                                1      13s

On the consumer side (Pod) we can use the Secret as usual as a volume:

volumes:
- name: p2c-sm-keys
  secret:
    secretName: pet2cattle-sm-keys

Or pushing the Secret as an environment variable:

env:
- name: DEMO_SECRET
  valueFrom:
    secretKeyRef:
      name: "pet2cattle-sm-keys"
      key: name_secret