6 min read | by Jordi Prats
To be able to modify a request to the Kubernetes API server prior to persist the object (to, for example, inject a sidecar) we can use a Mutating Webhook. The admission controller makes a requests using all the MutatingWebhookConfiguration objects that matches the request and processes them in serial:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
(...)
Let's take a look on how to configure a mutating webhook from scratch
First we will have to create an application to make the heavy lifting: It will receive the object that is going to add to the cluster on the /mutate URL:
@app.route('/mutate', methods=HTTP_METHODS)
def mutate():
(...)
It will send as a response a JSON-patch that will add the label powered-by to the pod:
@app.route('/mutate', methods=HTTP_METHODS)
def mutate():
(...)
try:
request_json = json.loads(request.data.decode('utf-8'))
except Exception as e:
print(str(e), flush = True)
print(str(request_json), flush = True)
patch = "[{ \"op\": \"add\", \"path\": \"/metadata/labels/powered-by\", \"value\": \"pet2cattle.com\" }]"
(...)
response = {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": request_json['request']['uid'],
"allowed": True,
'patch': patch_base64,
'patchType': "JSONPatch",
}
}
return json.dumps(response), 200, {'ContentType':'application/json-patch+json'}
Then we'll need to decide which URL it is going to be to generate the SSL certificates. If we create the following Kubernetes Service on the webhookdemo namespace:
kind: Service
apiVersion: v1
metadata:
name: pet2cattle-hook
spec:
selector:
component: pet2cattle-hook
ports:
- name: http
port: 443
targetPort: 8443
The URL the admission controller that will need to call is going to be pet2cattle-hook.webhookdemo.svc. It is going to expect to be a SSL certificate with a SAN record, we can create a new private key and the self-signed certificate as follows:
openssl req -new -sha256 \
-newkey rsa:2048 \
-subj "/C=RC/ST=Barcelona/O=pet2cattle/CN=pet2cattle-hook.webhookdemo.svc" \
-nodes -x509 \
-days 365 \
-out server.crt \
-addext "subjectAltName = DNS:pet2cattle-hook.webhookdemo.svc"
To be able to make it the certificate and the private key to the Pod we can use a ConfigMap mounted on /ssl as follows:
apiVersion: v1
kind: ConfigMap
metadata:
name: ssl-pet2cattle-webhook
data:
server.crt: |
-----BEGIN CERTIFICATE-----
MIIDzTCCArWgAwIBAgIUaQt6HUWmFD+MdQl8nWOyx+hlrMEwDQYJKoZIhvcNAQEL
BQAwYDELMAkGA1UEBhMCUkMxEjAQBgNVBAgMCUJhcmNlbG9uYTETMBEGA1UECgwK
cGV0MmNhdHRsZTEoMCYGA1UEAwwfcGV0MmNhdHRsZS1ob29rLndlYmhvb2tkZW1v
LnN2YzAeFw0yMTA4MTEyMDA2MzJaFw0yMjA4MTEyMDA2MzJaMGAxCzAJBgNVBAYT
AlJDMRIwEAYDVQQIDAlCYXJjZWxvbmExEzARBgNVBAoMCnBldDJjYXR0bGUxKDAm
BgNVBAMMH3BldDJjYXR0bGUtaG9vay53ZWJob29rZGVtby5zdmMwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC1mYz3tQdYciUzdm0kY22lGfhRLFs4y0Et
vgm0Icktt2k+pWhuG532lKFIjtmL0/G+lmPZUlRm7S7DUITy2yP2RK1zO5gUce2r
IOr4Ag5zWpYlYAUpKzPxQ3Igoi9l9tXa77Lsf0FXMLO47vfUEBDea+ekJZqjvxti
fyE1Xc7BLDENpRIv3GweXvVgBEtio1rCQ230vFuxGaHgQU3Qt28dyD8N1P2RbXQ4
M5D74ahBIfbCeyBvOmyv5Bm0BT6YJrEVAY62ZeyKCW6SNM5psbfnoBG4bO62sk0z
K+7zWWQm8/4S+k1FgKXt8ZuNhtRoBpATAK8mcp5c8F9w/7x1QG/FAgMBAAGjfzB9
MB0GA1UdDgQWBBTnfBu9wPj1XZeZGZ7tmQ8X7fsU8DAfBgNVHSMEGDAWgBTnfBu9
wPj1XZeZGZ7tmQ8X7fsU8DAPBgNVHRMBAf8EBTADAQH/MCoGA1UdEQQjMCGCH3Bl
dDJjYXR0bGUtaG9vay53ZWJob29rZGVtby5zdmMwDQYJKoZIhvcNAQELBQADggEB
AEf5FdToi9YURf5bf7NfVAU6l70RbFgfFWv4dzMFCp+jsgMOeL3O57IU80rRxHBI
FEkxIfIhiTnR+adDU+eiXm8OjM4bQUIlioMhoDBfuDkFvksWCDqRfR+3iBX7y7Qe
UDvGp+NBiBm7+kBuDG4xuP41r+GUuw59LLzNt0+kRQp6HxtzL+mJ1V8beWMrXFrU
M4ZwGdmTSb4swP+hC2wGPI5EOOsBkDc2eqXgRR2n63eSTVC8Eo0UWJQw/6RGuzBW
q4IyHUh5xoZ7Txea78truVgs413Duhdvp09Vn8GgUgRYXefA0hEM6RRO9sDLIK9f
UVIT2Fzb2xETIM3IMOjB6sU=
-----END CERTIFICATE-----
server.key: |
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC1mYz3tQdYciUz
dm0kY22lGfhRLFs4y0Etvgm0Icktt2k+pWhuG532lKFIjtmL0/G+lmPZUlRm7S7D
UITy2yP2RK1zO5gUce2rIOr4Ag5zWpYlYAUpKzPxQ3Igoi9l9tXa77Lsf0FXMLO4
7vfUEBDea+ekJZqjvxtifyE1Xc7BLDENpRIv3GweXvVgBEtio1rCQ230vFuxGaHg
QU3Qt28dyD8N1P2RbXQ4M5D74ahBIfbCeyBvOmyv5Bm0BT6YJrEVAY62ZeyKCW6S
NM5psbfnoBG4bO62sk0zK+7zWWQm8/4S+k1FgKXt8ZuNhtRoBpATAK8mcp5c8F9w
/7x1QG/FAgMBAAECggEAQ8ClImnM8serb3bYo4HhD38P8SEOa7MRf0JulmEgkMjk
IDZQLvxow+2R+uMo8Q1DHSs414Tq7nfBQaeR4pW15hSbbemnBMG4vWcLozoJMCp0
6D7ZzhFLUNEsDFbWPkGIaiWR6MBVnXUTKIUnu1u/H2y8wLYy6rLLQcVSm3mDQPhd
gYNOVPIMHF2Y4axYdlkuAnKjI4LDPJohFeT5qD/hCdP3s6M95CI6g9F+YGLIEwhD
9sj8b9//BEe/A94Dl6usIA2MHlp9e8vqapF/R0CfXSuZQXATri75U5JwkHTzYnpE
qgTnBb3ozJEA6L9zquMmrH7zG0ppaKAZnV9iIZSQhQKBgQDoaDgOeNZxmmKIfSHN
JWEfd9UA3C64d9imdupx9WnVtc2u1ul2t4S60okyB3b9piaS0fLHNL5B15IZRJxY
27fpDwvYG7azdWJISRbQD7CIoJo3Isixz4Xsq53W7kpMqdltDyp4iTLYYVEzyShV
CLEW4Cut71DbDDwEHYZKCAuq1wKBgQDICPRxzbJ9qE8qoTm6IWQ78Zu2f7xAl9sv
srDfl5FOIWlUXLTg848jBGZd41la1kex+KsTL6eTqA51CEKChsLbX+Ouq0iL0kpX
iqDb0wxkOdojMWuFZ4x0RSLmv9juuYzddpZvORMdBYD9mB6Vti8K/vJzSFl3UE/S
cqdiOuFiwwKBgQCQx9QcF+UnslCtzJ5RCXc+vk0gkwo7+tUppq0YvxTmgLKYt+OL
BHqYU+4KD6JuE6K2FjqTJOVdaSjnutlXddFVS/1J7MHdfEP02itvBEcqZjqMHIxA
URKSRLs4mQwKRElh6m+/1WCqcb2/cBJDHv4LTS2I1qxdOXrt6WKuHeL+0wKBgHFk
2iU1JMykv5P75zyDN03fzZRr3qyDKQZl9mwZgI5Y1Fu1XffzOZ3xHZJ1ka6zr9rM
izYKGqXSa7eeIg3aFNXFCs12XV6dq/TqKfvTLMAYJ3cxybDLHUy/8GP8Nx5E4vyb
//U21oXqG9AmDphxuUMzeP8u8UB4r3ct9YLyu9d/AoGBAIoAlWlwAsMjm0KTs2kt
7yyaLhxX55cCRnW1adedrqLPUXOZEkF7nHCOh4W6iYWnFvnZdGY5YqlPOVHoCsA3
xcKGxsOvoEZnDyNmRu8Y7i1Ta7iZ8BWq7hw0nNxNfVYGgw0UQaKDQZNx0V/BUP4y
6YmjXNyII6Av3qMqzqpIaNOI
-----END PRIVATE KEY-----
Then we will need to create a HTTPS enabled container to use these certificates, we can configure gunicorn to use the SSL certificates present on the /ssl directory:
FROM python:3.8-alpine
WORKDIR /code
# GUNICORN - not an actual dependency
RUN pip install gunicorn
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY app /code/app
EXPOSE 8443
USER 1001:1001
CMD [ "/usr/local/bin/gunicorn", "app:app", "--certfile=/ssl/server.crt", "--keyfile=/ssl/server.key", "--bind", "0.0.0.0:8443", "--keep-alive", "1" ]
To be able to have a pod that mounts the ConfigMap with the SSL certificates we will create a Deployment object as follows:
apiVersion: apps/v1
kind: Deployment
metadata:
name: pet2cattle-hook
spec:
replicas: 1
selector:
matchLabels:
component: pet2cattle-hook
template:
metadata:
labels:
component: pet2cattle-hook
spec:
containers:
- name: pet2cattle
image: jordiprats/pet2cattle-mutating-webhook:1
ports:
- name: http
containerPort: 8443
volumeMounts:
- name: ssl-pet2cattle-webhook
mountPath: /ssl
volumes:
- name: ssl-pet2cattle-webhook
configMap:
name: ssl-pet2cattle-webhook
Once this is in place we will have to create the actual MutatingWebhookConfiguration that will make Kubernetes make the request to the Deployment to retrieve the JSON-patch. On this object we will have to specify the CA, since we are using a self-signed vertificate we can just use the self-signed certificate itself in base64.
To get the base64 encoded string of the certificate file on a single file we can just disable wrapping using the -w 0 option like follows:
$ base64 -w0 server.crt
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR6VENDQXJXZ0F3SUJBZ0lVYVF0NkhVV21GRCtNZFFsOG5XT3l4K2hsck1Fd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lERUxNQWtHQTFVRUJoTUNVa014RWpBUUJnTlZCQWdNQ1VKaGNtTmxiRzl1WVRFVE1CRUdBMVVFQ2d3SwpjR1YwTW1OaGRIUnNaVEVvTUNZR0ExVUVBd3dmY0dWME1tTmhkSFJzWlMxb2IyOXJMbmRsWW1odmIydGtaVzF2CkxuTjJZekFlRncweU1UQTRNVEV5TURBMk16SmFGdzB5TWpBNE1URXlNREEyTXpKYU1HQXhDekFKQmdOVkJBWVQKQWxKRE1SSXdFQVlEVlFRSURBbENZWEpqWld4dmJtRXhFekFSQmdOVkJBb01DbkJsZERKallYUjBiR1V4S0RBbQpCZ05WQkFNTUgzQmxkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUMxbVl6M3RRZFljaVV6ZG0wa1kyMmxHZmhSTEZzNHkwRXQKdmdtMElja3R0MmsrcFdodUc1MzJsS0ZJanRtTDAvRytsbVBaVWxSbTdTN0RVSVR5MnlQMlJLMXpPNWdVY2UycgpJT3I0QWc1eldwWWxZQVVwS3pQeFEzSWdvaTlsOXRYYTc3THNmMEZYTUxPNDd2ZlVFQkRlYStla0pacWp2eHRpCmZ5RTFYYzdCTERFTnBSSXYzR3dlWHZWZ0JFdGlvMXJDUTIzMHZGdXhHYUhnUVUzUXQyOGR5RDhOMVAyUmJYUTQKTTVENzRhaEJJZmJDZXlCdk9teXY1Qm0wQlQ2WUpyRVZBWTYyWmV5S0NXNlNOTTVwc2Jmbm9CRzRiTzYyc2swegpLKzd6V1dRbTgvNFMrazFGZ0tYdDhadU5odFJvQnBBVEFLOG1jcDVjOEY5dy83eDFRRy9GQWdNQkFBR2pmekI5Ck1CMEdBMVVkRGdRV0JCVG5mQnU5d1BqMVhaZVpHWjd0bVE4WDdmc1U4REFmQmdOVkhTTUVHREFXZ0JUbmZCdTkKd1BqMVhaZVpHWjd0bVE4WDdmc1U4REFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQ29HQTFVZEVRUWpNQ0dDSDNCbApkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCkFFZjVGZFRvaTlZVVJmNWJmN05mVkFVNmw3MFJiRmdmRld2NGR6TUZDcCtqc2dNT2VMM081N0lVODByUnhIQkkKRkVreElmSWhpVG5SK2FkRFUrZWlYbThPak00YlFVSWxpb01ob0RCZnVEa0Z2a3NXQ0RxUmZSKzNpQlg3eTdRZQpVRHZHcCtOQmlCbTcra0J1REc0eHVQNDFyK0dVdXc1OUxMek50MCtrUlFwNkh4dHpMK21KMVY4YmVXTXJYRnJVCk00WndHZG1UU2I0c3dQK2hDMndHUEk1RU9Pc0JrRGMyZXFYZ1JSMm42M2VTVFZDOEVvMFVXSlF3LzZSR3V6QlcKcTRJeUhVaDV4b1o3VHhlYTc4dHJ1VmdzNDEzRHVoZHZwMDlWbjhHZ1VnUllYZWZBMGhFTTZSUk85c0RMSUs5ZgpVVklUMkZ6YjJ4RVRJTTNJTU9qQjZzVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
This output needs to to to as a string on the caBundle key:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: pet2cattle-webhook
webhooks:
- name: pet2cattle-hook.webhookdemo.svc
failurePolicy: Fail
clientConfig:
service:
name: pet2cattle-hook
namespace: webhookdemo
path: "/mutate"
caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR6VENDQXJXZ0F3SUJBZ0lVYVF0NkhVV21GRCtNZFFsOG5XT3l4K2hsck1Fd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lERUxNQWtHQTFVRUJoTUNVa014RWpBUUJnTlZCQWdNQ1VKaGNtTmxiRzl1WVRFVE1CRUdBMVVFQ2d3SwpjR1YwTW1OaGRIUnNaVEVvTUNZR0ExVUVBd3dmY0dWME1tTmhkSFJzWlMxb2IyOXJMbmRsWW1odmIydGtaVzF2CkxuTjJZekFlRncweU1UQTRNVEV5TURBMk16SmFGdzB5TWpBNE1URXlNREEyTXpKYU1HQXhDekFKQmdOVkJBWVQKQWxKRE1SSXdFQVlEVlFRSURBbENZWEpqWld4dmJtRXhFekFSQmdOVkJBb01DbkJsZERKallYUjBiR1V4S0RBbQpCZ05WQkFNTUgzQmxkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUMxbVl6M3RRZFljaVV6ZG0wa1kyMmxHZmhSTEZzNHkwRXQKdmdtMElja3R0MmsrcFdodUc1MzJsS0ZJanRtTDAvRytsbVBaVWxSbTdTN0RVSVR5MnlQMlJLMXpPNWdVY2UycgpJT3I0QWc1eldwWWxZQVVwS3pQeFEzSWdvaTlsOXRYYTc3THNmMEZYTUxPNDd2ZlVFQkRlYStla0pacWp2eHRpCmZ5RTFYYzdCTERFTnBSSXYzR3dlWHZWZ0JFdGlvMXJDUTIzMHZGdXhHYUhnUVUzUXQyOGR5RDhOMVAyUmJYUTQKTTVENzRhaEJJZmJDZXlCdk9teXY1Qm0wQlQ2WUpyRVZBWTYyWmV5S0NXNlNOTTVwc2Jmbm9CRzRiTzYyc2swegpLKzd6V1dRbTgvNFMrazFGZ0tYdDhadU5odFJvQnBBVEFLOG1jcDVjOEY5dy83eDFRRy9GQWdNQkFBR2pmekI5Ck1CMEdBMVVkRGdRV0JCVG5mQnU5d1BqMVhaZVpHWjd0bVE4WDdmc1U4REFmQmdOVkhTTUVHREFXZ0JUbmZCdTkKd1BqMVhaZVpHWjd0bVE4WDdmc1U4REFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQ29HQTFVZEVRUWpNQ0dDSDNCbApkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCkFFZjVGZFRvaTlZVVJmNWJmN05mVkFVNmw3MFJiRmdmRld2NGR6TUZDcCtqc2dNT2VMM081N0lVODByUnhIQkkKRkVreElmSWhpVG5SK2FkRFUrZWlYbThPak00YlFVSWxpb01ob0RCZnVEa0Z2a3NXQ0RxUmZSKzNpQlg3eTdRZQpVRHZHcCtOQmlCbTcra0J1REc0eHVQNDFyK0dVdXc1OUxMek50MCtrUlFwNkh4dHpMK21KMVY4YmVXTXJYRnJVCk00WndHZG1UU2I0c3dQK2hDMndHUEk1RU9Pc0JrRGMyZXFYZ1JSMm42M2VTVFZDOEVvMFVXSlF3LzZSR3V6QlcKcTRJeUhVaDV4b1o3VHhlYTc4dHJ1VmdzNDEzRHVoZHZwMDlWbjhHZ1VnUllYZWZBMGhFTTZSUk85c0RMSUs5ZgpVVklUMkZ6YjJ4RVRJTTNJTU9qQjZzVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
rules:
- operations: [ "CREATE" ]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
admissionReviewVersions: ["v1", "v1beta1"]
sideEffects: None
timeoutSeconds: 5
Finally, we are going to create the webhookdemo namespace as defined on the MutatingWebhookConfiguration:
$ kubectl create ns webhookdemo
namespace/webhookdemo created
$ kubectl config set-context --current --namespace webhookdemo
Context "minikube" modified.
Finally, I've added a yaml file with all the demo objects on the kubernetes-mutating-webhook repository to make it easier to create:
$ kubectl apply -f yaml/demo-webhook.yaml
configmap/ssl-pet2cattle-webhook created
deployment.apps/pet2cattle-hook created
service/pet2cattle-hook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/pet2cattle-webhook created
We will have to wait for the Pod to be in Running state:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pet2cattle-hook-8cff54d84-gq487 0/1 ContainerCreating 0 6s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pet2cattle-hook-8cff54d84-gq487 1/1 Running 0 12s
Once it is ready, any new pod will contain the powered-by label. On the kubernetes-mutating-webhook repository I have also included a test Pod:
$ kubectl apply -f tests/testpod.yaml
pod/test-mutator-webhook created
If we describe it we will be able to see how the label have been added without being present on the yaml configuration:
$ kubectl describe pod/test-mutator-webhook
Name: test-mutator-webhook
Namespace: webhookdemo
Priority: 0
Node: minikube/192.168.49.2
Start Time: Wed, 11 Aug 2021 22:16:58 +0200
Labels: powered-by=pet2cattle.com
test=test-mutator-webhook
Annotations: <none>
Status: Running
IP: 172.17.0.5
IPs:
IP: 172.17.0.5
Containers:
demo:
Container ID: docker://c25b464fca1392d23bda538517bdfffdfcd8a769dcf1049e7b9c037b7befeee9
Image: nginx
Image ID: docker-pullable://nginx@sha256:8f335768880da6baf72b70c701002b45f4932acae8d574dedfddaf967fc3ac90
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 11 Aug 2021 22:17:20 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lf94n (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-lf94n:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 63s default-scheduler Successfully assigned webhookdemo/test-mutator-webhook to minikube
Normal Pulling 59s kubelet Pulling image "nginx"
Normal Pulled 45s kubelet Successfully pulled image "nginx" in 14.13825962s
Normal Created 41s kubelet Created container demo
Normal Started 40s kubelet Started container demo
Do we really need to go into all that trouble to modify Kubernetes objects as they are created? Not really, there are generic tools to do so like the KubeMod operator
Posted on 12/08/2021