Kubernetes: How to identify when and how an object has been modified

3 min read | by Jordi Prats

In the same way we can use git blame to identify who and when has modified a specific line, with kubectl blame we'll be able to do the same to Kubernetes objects

It is a krew plugin so installing it is as easy as follows:

$ kubectl krew install blame

Once installed we just need to specify the object we want to see:

$ kubectl blame hpa ampa-core
                                              apiVersion: autoscaling/v1
                                              kind: HorizontalPodAutoscaler
                                              metadata:
Go-http-client          (Update 18 hours ago)   annotations:
kube-controller-manager (Update 18 hours ago)     autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2022-05-12T23:14:33Z","reason":"ScaleDownStabilized","message":"recent
  recommendations were higher than current one, applying the highest recent recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2022-05-13T05:18:05Z","reason":"ValidMetricFound","message":"the
  HPA was able to successfully calculate a replica count from cpu resource utilization
  (percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2022-05-13T05:18:05Z","reason":"DesiredWithinRange","message":"the
  desired count is within the acceptable range"}]'
kube-controller-manager (Update 18 hours ago)     autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":2,"currentAverageValue":"22m"}}]'
Go-http-client          (Update 18 hours ago)     helm.sh/hook: post-install,post-upgrade
Go-http-client          (Update 18 hours ago)     helm.sh/hook-weight: "100"
                                                creationTimestamp: "2022-05-12T23:14:30Z"
                                                name: ampa-core
                                                namespace: pet2cattle
                                                resourceVersion: "138072472"
                                                uid: 4bc1d4b0-4e8d-4ecb-b7df-6951c7db6b04
                                              spec:
Go-http-client          (Update 18 hours ago)   maxReplicas: 2
Go-http-client          (Update 18 hours ago)   minReplicas: 2
                                                scaleTargetRef:
Go-http-client          (Update 18 hours ago)     apiVersion: apps/v1
Go-http-client          (Update 18 hours ago)     kind: Deployment
Go-http-client          (Update 18 hours ago)     name: ampa-core
Go-http-client          (Update 18 hours ago)   targetCPUUtilizationPercentage: 70
                                              status:
kube-controller-manager (Update 18 hours ago)   currentCPUUtilizationPercentage: 2
kube-controller-manager (Update 18 hours ago)   currentReplicas: 2
kube-controller-manager (Update 18 hours ago)   desiredReplicas: 2

For example, if we edit the object with kubectl edit:

$ kubectl edit hpa ampa-core
horizontalpodautoscaler.autoscaling/ampa-core edited

And repeat the kubectl krew command we'll be able to see the difference:

$ kubectl blame hpa ampa-core
                                               apiVersion: autoscaling/v1
                                               kind: HorizontalPodAutoscaler
                                               metadata:
Go-http-client          (Update  18 hours ago)   annotations:
kube-controller-manager (Update  18 hours ago)     autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2022-05-12T23:14:33Z","reason":"ScaleDownStabilized","message":"recent
  recommendations were higher than current one, applying the highest recent recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2022-05-13T05:18:05Z","reason":"ValidMetricFound","message":"the
  HPA was able to successfully calculate a replica count from cpu resource utilization
  (percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2022-05-13T05:18:05Z","reason":"DesiredWithinRange","message":"the
  desired count is within the acceptable range"}]'
kube-controller-manager (Update  18 hours ago)     autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":2,"currentAverageValue":"21m"}}]'
Go-http-client          (Update  18 hours ago)     helm.sh/hook: post-install,post-upgrade
Go-http-client          (Update  18 hours ago)     helm.sh/hook-weight: "100"
                                                 creationTimestamp: "2022-05-12T23:14:30Z"
                                                 name: ampa-core
                                                 namespace: pet2cattle
                                                 resourceVersion: "138072984"
                                                 uid: 4bc1d4b0-4e8d-4ecb-b7df-6951c7db6b04
                                               spec:
Go-http-client          (Update  18 hours ago)   maxReplicas: 2
kubectl-edit            (Update 5 seconds ago)   minReplicas: 1
                                                 scaleTargetRef:
Go-http-client          (Update  18 hours ago)     apiVersion: apps/v1
Go-http-client          (Update  18 hours ago)     kind: Deployment
Go-http-client          (Update  18 hours ago)     name: ampa-core
Go-http-client          (Update  18 hours ago)   targetCPUUtilizationPercentage: 70
                                               status:
kube-controller-manager (Update  18 hours ago)   currentCPUUtilizationPercentage: 2
kube-controller-manager (Update  18 hours ago)   currentReplicas: 2
kube-controller-manager (Update  18 hours ago)   desiredReplicas: 2

It uses the .metadata.manageFields to highlight the changes:

$ kubectl get hpa spin-orca -o yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  managedFields:
  - apiVersion: autoscaling/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:helm.sh/hook: {}
          f:helm.sh/hook-weight: {}
      f:spec:
        f:maxReplicas: {}
        f:scaleTargetRef:
          f:apiVersion: {}
          f:kind: {}
          f:name: {}
        f:targetCPUUtilizationPercentage: {}
    manager: Go-http-client
    operation: Update
    time: "2022-05-12T23:14:30Z"
  - apiVersion: autoscaling/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:minReplicas: {}
    manager: kubectl-edit
    operation: Update
    time: "2022-05-13T05:21:38Z"
  - apiVersion: autoscaling/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:autoscaling.alpha.kubernetes.io/conditions: {}
          f:autoscaling.alpha.kubernetes.io/current-metrics: {}
      f:status:
        f:currentCPUUtilizationPercentage: {}
        f:currentReplicas: {}
        f:desiredReplicas: {}
        f:lastScaleTime: {}
    manager: kube-controller-manager
    operation: Update
    time: "2022-05-13T05:23:05Z"
  name: ampa-core
  namespace: pet2cattle
  resourceVersion: "138075242"
  uid: 4bc1d4b0-4e8d-4ecb-b7df-6951c7db6b04
(...)

Posted on 13/05/2022