Kubernetes Mutating Webhook: Patch a Kubernetes Pod on the fly - the hard way

6 min read | by Jordi Prats

To be able to modify a request to the Kubernetes API server prior to persist the object (to, for example, inject a sidecar) we can use a Mutating Webhook. The admission controller makes a requests using all the MutatingWebhookConfiguration objects that matches the request and processes them in serial:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
(...)

Let's take a look on how to configure a mutating webhook from scratch

First we will have to create an application to make the heavy lifting: It will receive the object that is going to add to the cluster on the /mutate URL:

@app.route('/mutate', methods=HTTP_METHODS)
def mutate():
(...)

It will send as a response a JSON-patch that will add the label powered-by to the pod:

@app.route('/mutate', methods=HTTP_METHODS)
def mutate():
  (...)

  try:
    request_json = json.loads(request.data.decode('utf-8'))
  except Exception as e:
    print(str(e), flush = True)

  print(str(request_json), flush = True)

  patch = "[{ \"op\": \"add\", \"path\": \"/metadata/labels/powered-by\", \"value\": \"pet2cattle.com\" }]"

  (...)

  response = {
    "apiVersion": "admission.k8s.io/v1",
    "kind": "AdmissionReview",
    "response": {
      "uid": request_json['request']['uid'],
      "allowed": True,
      'patch': patch_base64,
      'patchType': "JSONPatch",
    }
  }

  return json.dumps(response), 200, {'ContentType':'application/json-patch+json'} 

Then we'll need to decide which URL it is going to be to generate the SSL certificates. If we create the following Kubernetes Service on the webhookdemo namespace:

kind: Service
apiVersion: v1
metadata:
  name: pet2cattle-hook
spec:
  selector:
    component: pet2cattle-hook
  ports:
  - name: http
    port: 443
    targetPort: 8443

The URL the admission controller that will need to call is going to be pet2cattle-hook.webhookdemo.svc. It is going to expect to be a SSL certificate with a SAN record, we can create a new private key and the self-signed certificate as follows:

openssl req -new -sha256 \
    -newkey rsa:2048 \
    -subj "/C=RC/ST=Barcelona/O=pet2cattle/CN=pet2cattle-hook.webhookdemo.svc" \
    -nodes -x509 \
    -days 365 \
    -out server.crt \
    -addext "subjectAltName = DNS:pet2cattle-hook.webhookdemo.svc"

To be able to make it the certificate and the private key to the Pod we can use a ConfigMap mounted on /ssl as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ssl-pet2cattle-webhook
data:
  server.crt: |
    -----BEGIN CERTIFICATE-----
    MIIDzTCCArWgAwIBAgIUaQt6HUWmFD+MdQl8nWOyx+hlrMEwDQYJKoZIhvcNAQEL
    BQAwYDELMAkGA1UEBhMCUkMxEjAQBgNVBAgMCUJhcmNlbG9uYTETMBEGA1UECgwK
    cGV0MmNhdHRsZTEoMCYGA1UEAwwfcGV0MmNhdHRsZS1ob29rLndlYmhvb2tkZW1v
    LnN2YzAeFw0yMTA4MTEyMDA2MzJaFw0yMjA4MTEyMDA2MzJaMGAxCzAJBgNVBAYT
    AlJDMRIwEAYDVQQIDAlCYXJjZWxvbmExEzARBgNVBAoMCnBldDJjYXR0bGUxKDAm
    BgNVBAMMH3BldDJjYXR0bGUtaG9vay53ZWJob29rZGVtby5zdmMwggEiMA0GCSqG
    SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC1mYz3tQdYciUzdm0kY22lGfhRLFs4y0Et
    vgm0Icktt2k+pWhuG532lKFIjtmL0/G+lmPZUlRm7S7DUITy2yP2RK1zO5gUce2r
    IOr4Ag5zWpYlYAUpKzPxQ3Igoi9l9tXa77Lsf0FXMLO47vfUEBDea+ekJZqjvxti
    fyE1Xc7BLDENpRIv3GweXvVgBEtio1rCQ230vFuxGaHgQU3Qt28dyD8N1P2RbXQ4
    M5D74ahBIfbCeyBvOmyv5Bm0BT6YJrEVAY62ZeyKCW6SNM5psbfnoBG4bO62sk0z
    K+7zWWQm8/4S+k1FgKXt8ZuNhtRoBpATAK8mcp5c8F9w/7x1QG/FAgMBAAGjfzB9
    MB0GA1UdDgQWBBTnfBu9wPj1XZeZGZ7tmQ8X7fsU8DAfBgNVHSMEGDAWgBTnfBu9
    wPj1XZeZGZ7tmQ8X7fsU8DAPBgNVHRMBAf8EBTADAQH/MCoGA1UdEQQjMCGCH3Bl
    dDJjYXR0bGUtaG9vay53ZWJob29rZGVtby5zdmMwDQYJKoZIhvcNAQELBQADggEB
    AEf5FdToi9YURf5bf7NfVAU6l70RbFgfFWv4dzMFCp+jsgMOeL3O57IU80rRxHBI
    FEkxIfIhiTnR+adDU+eiXm8OjM4bQUIlioMhoDBfuDkFvksWCDqRfR+3iBX7y7Qe
    UDvGp+NBiBm7+kBuDG4xuP41r+GUuw59LLzNt0+kRQp6HxtzL+mJ1V8beWMrXFrU
    M4ZwGdmTSb4swP+hC2wGPI5EOOsBkDc2eqXgRR2n63eSTVC8Eo0UWJQw/6RGuzBW
    q4IyHUh5xoZ7Txea78truVgs413Duhdvp09Vn8GgUgRYXefA0hEM6RRO9sDLIK9f
    UVIT2Fzb2xETIM3IMOjB6sU=
    -----END CERTIFICATE-----
  server.key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC1mYz3tQdYciUz
    dm0kY22lGfhRLFs4y0Etvgm0Icktt2k+pWhuG532lKFIjtmL0/G+lmPZUlRm7S7D
    UITy2yP2RK1zO5gUce2rIOr4Ag5zWpYlYAUpKzPxQ3Igoi9l9tXa77Lsf0FXMLO4
    7vfUEBDea+ekJZqjvxtifyE1Xc7BLDENpRIv3GweXvVgBEtio1rCQ230vFuxGaHg
    QU3Qt28dyD8N1P2RbXQ4M5D74ahBIfbCeyBvOmyv5Bm0BT6YJrEVAY62ZeyKCW6S
    NM5psbfnoBG4bO62sk0zK+7zWWQm8/4S+k1FgKXt8ZuNhtRoBpATAK8mcp5c8F9w
    /7x1QG/FAgMBAAECggEAQ8ClImnM8serb3bYo4HhD38P8SEOa7MRf0JulmEgkMjk
    IDZQLvxow+2R+uMo8Q1DHSs414Tq7nfBQaeR4pW15hSbbemnBMG4vWcLozoJMCp0
    6D7ZzhFLUNEsDFbWPkGIaiWR6MBVnXUTKIUnu1u/H2y8wLYy6rLLQcVSm3mDQPhd
    gYNOVPIMHF2Y4axYdlkuAnKjI4LDPJohFeT5qD/hCdP3s6M95CI6g9F+YGLIEwhD
    9sj8b9//BEe/A94Dl6usIA2MHlp9e8vqapF/R0CfXSuZQXATri75U5JwkHTzYnpE
    qgTnBb3ozJEA6L9zquMmrH7zG0ppaKAZnV9iIZSQhQKBgQDoaDgOeNZxmmKIfSHN
    JWEfd9UA3C64d9imdupx9WnVtc2u1ul2t4S60okyB3b9piaS0fLHNL5B15IZRJxY
    27fpDwvYG7azdWJISRbQD7CIoJo3Isixz4Xsq53W7kpMqdltDyp4iTLYYVEzyShV
    CLEW4Cut71DbDDwEHYZKCAuq1wKBgQDICPRxzbJ9qE8qoTm6IWQ78Zu2f7xAl9sv
    srDfl5FOIWlUXLTg848jBGZd41la1kex+KsTL6eTqA51CEKChsLbX+Ouq0iL0kpX
    iqDb0wxkOdojMWuFZ4x0RSLmv9juuYzddpZvORMdBYD9mB6Vti8K/vJzSFl3UE/S
    cqdiOuFiwwKBgQCQx9QcF+UnslCtzJ5RCXc+vk0gkwo7+tUppq0YvxTmgLKYt+OL
    BHqYU+4KD6JuE6K2FjqTJOVdaSjnutlXddFVS/1J7MHdfEP02itvBEcqZjqMHIxA
    URKSRLs4mQwKRElh6m+/1WCqcb2/cBJDHv4LTS2I1qxdOXrt6WKuHeL+0wKBgHFk
    2iU1JMykv5P75zyDN03fzZRr3qyDKQZl9mwZgI5Y1Fu1XffzOZ3xHZJ1ka6zr9rM
    izYKGqXSa7eeIg3aFNXFCs12XV6dq/TqKfvTLMAYJ3cxybDLHUy/8GP8Nx5E4vyb
    //U21oXqG9AmDphxuUMzeP8u8UB4r3ct9YLyu9d/AoGBAIoAlWlwAsMjm0KTs2kt
    7yyaLhxX55cCRnW1adedrqLPUXOZEkF7nHCOh4W6iYWnFvnZdGY5YqlPOVHoCsA3
    xcKGxsOvoEZnDyNmRu8Y7i1Ta7iZ8BWq7hw0nNxNfVYGgw0UQaKDQZNx0V/BUP4y
    6YmjXNyII6Av3qMqzqpIaNOI
    -----END PRIVATE KEY-----

Then we will need to create a HTTPS enabled container to use these certificates, we can configure gunicorn to use the SSL certificates present on the /ssl directory:

FROM python:3.8-alpine

WORKDIR /code

# GUNICORN - not an actual dependency
RUN pip install gunicorn

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY app /code/app

EXPOSE 8443

USER 1001:1001

CMD [ "/usr/local/bin/gunicorn", "app:app", "--certfile=/ssl/server.crt", "--keyfile=/ssl/server.key", "--bind", "0.0.0.0:8443", "--keep-alive", "1" ]

To be able to have a pod that mounts the ConfigMap with the SSL certificates we will create a Deployment object as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pet2cattle-hook
spec:
  replicas: 1
  selector:
    matchLabels:
      component: pet2cattle-hook
  template:
    metadata:
      labels:
        component: pet2cattle-hook
    spec:
      containers:
      - name: pet2cattle
        image: jordiprats/pet2cattle-mutating-webhook:1
        ports:
        - name: http
          containerPort: 8443
        volumeMounts:
        - name: ssl-pet2cattle-webhook
          mountPath: /ssl
      volumes:
      - name: ssl-pet2cattle-webhook
        configMap:
          name: ssl-pet2cattle-webhook

Once this is in place we will have to create the actual MutatingWebhookConfiguration that will make Kubernetes make the request to the Deployment to retrieve the JSON-patch. On this object we will have to specify the CA, since we are using a self-signed vertificate we can just use the self-signed certificate itself in base64.

To get the base64 encoded string of the certificate file on a single file we can just disable wrapping using the -w 0 option like follows:

$ base64 -w0 server.crt 
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR6VENDQXJXZ0F3SUJBZ0lVYVF0NkhVV21GRCtNZFFsOG5XT3l4K2hsck1Fd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lERUxNQWtHQTFVRUJoTUNVa014RWpBUUJnTlZCQWdNQ1VKaGNtTmxiRzl1WVRFVE1CRUdBMVVFQ2d3SwpjR1YwTW1OaGRIUnNaVEVvTUNZR0ExVUVBd3dmY0dWME1tTmhkSFJzWlMxb2IyOXJMbmRsWW1odmIydGtaVzF2CkxuTjJZekFlRncweU1UQTRNVEV5TURBMk16SmFGdzB5TWpBNE1URXlNREEyTXpKYU1HQXhDekFKQmdOVkJBWVQKQWxKRE1SSXdFQVlEVlFRSURBbENZWEpqWld4dmJtRXhFekFSQmdOVkJBb01DbkJsZERKallYUjBiR1V4S0RBbQpCZ05WQkFNTUgzQmxkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUMxbVl6M3RRZFljaVV6ZG0wa1kyMmxHZmhSTEZzNHkwRXQKdmdtMElja3R0MmsrcFdodUc1MzJsS0ZJanRtTDAvRytsbVBaVWxSbTdTN0RVSVR5MnlQMlJLMXpPNWdVY2UycgpJT3I0QWc1eldwWWxZQVVwS3pQeFEzSWdvaTlsOXRYYTc3THNmMEZYTUxPNDd2ZlVFQkRlYStla0pacWp2eHRpCmZ5RTFYYzdCTERFTnBSSXYzR3dlWHZWZ0JFdGlvMXJDUTIzMHZGdXhHYUhnUVUzUXQyOGR5RDhOMVAyUmJYUTQKTTVENzRhaEJJZmJDZXlCdk9teXY1Qm0wQlQ2WUpyRVZBWTYyWmV5S0NXNlNOTTVwc2Jmbm9CRzRiTzYyc2swegpLKzd6V1dRbTgvNFMrazFGZ0tYdDhadU5odFJvQnBBVEFLOG1jcDVjOEY5dy83eDFRRy9GQWdNQkFBR2pmekI5Ck1CMEdBMVVkRGdRV0JCVG5mQnU5d1BqMVhaZVpHWjd0bVE4WDdmc1U4REFmQmdOVkhTTUVHREFXZ0JUbmZCdTkKd1BqMVhaZVpHWjd0bVE4WDdmc1U4REFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQ29HQTFVZEVRUWpNQ0dDSDNCbApkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCkFFZjVGZFRvaTlZVVJmNWJmN05mVkFVNmw3MFJiRmdmRld2NGR6TUZDcCtqc2dNT2VMM081N0lVODByUnhIQkkKRkVreElmSWhpVG5SK2FkRFUrZWlYbThPak00YlFVSWxpb01ob0RCZnVEa0Z2a3NXQ0RxUmZSKzNpQlg3eTdRZQpVRHZHcCtOQmlCbTcra0J1REc0eHVQNDFyK0dVdXc1OUxMek50MCtrUlFwNkh4dHpMK21KMVY4YmVXTXJYRnJVCk00WndHZG1UU2I0c3dQK2hDMndHUEk1RU9Pc0JrRGMyZXFYZ1JSMm42M2VTVFZDOEVvMFVXSlF3LzZSR3V6QlcKcTRJeUhVaDV4b1o3VHhlYTc4dHJ1VmdzNDEzRHVoZHZwMDlWbjhHZ1VnUllYZWZBMGhFTTZSUk85c0RMSUs5ZgpVVklUMkZ6YjJ4RVRJTTNJTU9qQjZzVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

This output needs to to to as a string on the caBundle key:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: pet2cattle-webhook
webhooks:
  - name: pet2cattle-hook.webhookdemo.svc
    failurePolicy: Fail
    clientConfig:
      service:
        name: pet2cattle-hook
        namespace: webhookdemo
        path: "/mutate"
      caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR6VENDQXJXZ0F3SUJBZ0lVYVF0NkhVV21GRCtNZFFsOG5XT3l4K2hsck1Fd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lERUxNQWtHQTFVRUJoTUNVa014RWpBUUJnTlZCQWdNQ1VKaGNtTmxiRzl1WVRFVE1CRUdBMVVFQ2d3SwpjR1YwTW1OaGRIUnNaVEVvTUNZR0ExVUVBd3dmY0dWME1tTmhkSFJzWlMxb2IyOXJMbmRsWW1odmIydGtaVzF2CkxuTjJZekFlRncweU1UQTRNVEV5TURBMk16SmFGdzB5TWpBNE1URXlNREEyTXpKYU1HQXhDekFKQmdOVkJBWVQKQWxKRE1SSXdFQVlEVlFRSURBbENZWEpqWld4dmJtRXhFekFSQmdOVkJBb01DbkJsZERKallYUjBiR1V4S0RBbQpCZ05WQkFNTUgzQmxkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUMxbVl6M3RRZFljaVV6ZG0wa1kyMmxHZmhSTEZzNHkwRXQKdmdtMElja3R0MmsrcFdodUc1MzJsS0ZJanRtTDAvRytsbVBaVWxSbTdTN0RVSVR5MnlQMlJLMXpPNWdVY2UycgpJT3I0QWc1eldwWWxZQVVwS3pQeFEzSWdvaTlsOXRYYTc3THNmMEZYTUxPNDd2ZlVFQkRlYStla0pacWp2eHRpCmZ5RTFYYzdCTERFTnBSSXYzR3dlWHZWZ0JFdGlvMXJDUTIzMHZGdXhHYUhnUVUzUXQyOGR5RDhOMVAyUmJYUTQKTTVENzRhaEJJZmJDZXlCdk9teXY1Qm0wQlQ2WUpyRVZBWTYyWmV5S0NXNlNOTTVwc2Jmbm9CRzRiTzYyc2swegpLKzd6V1dRbTgvNFMrazFGZ0tYdDhadU5odFJvQnBBVEFLOG1jcDVjOEY5dy83eDFRRy9GQWdNQkFBR2pmekI5Ck1CMEdBMVVkRGdRV0JCVG5mQnU5d1BqMVhaZVpHWjd0bVE4WDdmc1U4REFmQmdOVkhTTUVHREFXZ0JUbmZCdTkKd1BqMVhaZVpHWjd0bVE4WDdmc1U4REFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQ29HQTFVZEVRUWpNQ0dDSDNCbApkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCkFFZjVGZFRvaTlZVVJmNWJmN05mVkFVNmw3MFJiRmdmRld2NGR6TUZDcCtqc2dNT2VMM081N0lVODByUnhIQkkKRkVreElmSWhpVG5SK2FkRFUrZWlYbThPak00YlFVSWxpb01ob0RCZnVEa0Z2a3NXQ0RxUmZSKzNpQlg3eTdRZQpVRHZHcCtOQmlCbTcra0J1REc0eHVQNDFyK0dVdXc1OUxMek50MCtrUlFwNkh4dHpMK21KMVY4YmVXTXJYRnJVCk00WndHZG1UU2I0c3dQK2hDMndHUEk1RU9Pc0JrRGMyZXFYZ1JSMm42M2VTVFZDOEVvMFVXSlF3LzZSR3V6QlcKcTRJeUhVaDV4b1o3VHhlYTc4dHJ1VmdzNDEzRHVoZHZwMDlWbjhHZ1VnUllYZWZBMGhFTTZSUk85c0RMSUs5ZgpVVklUMkZ6YjJ4RVRJTTNJTU9qQjZzVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    admissionReviewVersions: ["v1", "v1beta1"]
    sideEffects: None
    timeoutSeconds: 5

Finally, we are going to create the webhookdemo namespace as defined on the MutatingWebhookConfiguration:

$ kubectl create ns webhookdemo
namespace/webhookdemo created
$ kubectl config set-context --current --namespace webhookdemo
Context "minikube" modified.

Finally, I've added a yaml file with all the demo objects on the kubernetes-mutating-webhook repository to make it easier to create:

$ kubectl apply -f yaml/demo-webhook.yaml 
configmap/ssl-pet2cattle-webhook created
deployment.apps/pet2cattle-hook created
service/pet2cattle-hook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/pet2cattle-webhook created

We will have to wait for the Pod to be in Running state:

$ kubectl get pods
NAME                              READY   STATUS              RESTARTS   AGE
pet2cattle-hook-8cff54d84-gq487   0/1     ContainerCreating   0          6s
$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
pet2cattle-hook-8cff54d84-gq487   1/1     Running   0          12s

Once it is ready, any new pod will contain the powered-by label. On the kubernetes-mutating-webhook repository I have also included a test Pod:

$ kubectl apply -f tests/testpod.yaml 
pod/test-mutator-webhook created

If we describe it we will be able to see how the label have been added without being present on the yaml configuration:

$ kubectl describe pod/test-mutator-webhook 
Name:         test-mutator-webhook
Namespace:    webhookdemo
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Wed, 11 Aug 2021 22:16:58 +0200
Labels:       powered-by=pet2cattle.com
              test=test-mutator-webhook
Annotations:  <none>
Status:       Running
IP:           172.17.0.5
IPs:
  IP:  172.17.0.5
Containers:
  demo:
    Container ID:   docker://c25b464fca1392d23bda538517bdfffdfcd8a769dcf1049e7b9c037b7befeee9
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:8f335768880da6baf72b70c701002b45f4932acae8d574dedfddaf967fc3ac90
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 11 Aug 2021 22:17:20 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lf94n (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-lf94n:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  63s   default-scheduler  Successfully assigned webhookdemo/test-mutator-webhook to minikube
  Normal  Pulling    59s   kubelet            Pulling image "nginx"
  Normal  Pulled     45s   kubelet            Successfully pulled image "nginx" in 14.13825962s
  Normal  Created    41s   kubelet            Created container demo
  Normal  Started    40s   kubelet            Started container demo

Do we really need to go into all that trouble to modify Kubernetes objects as they are created? Not really, there are generic tools to do so like the KubeMod operator


Posted on 12/08/2021