Autoscaling using datadog as a external metrics provider

kubernetes hpa datadog external metrics

3 min read | by Jordi Prats

Since Kubernetes v1.2 we can autoscale an application based on metrics like CPU provided by the metrics-server. As of Kubernetes v1.6, it is possible to autoscale off of custom metrics and later on, starting Kubernetes v1.10, we can autoscale using any metric from outside the cluster, like the ones collected by datadog

To be able to configure an HPA object to use datadog's metrics we will have to enable the metrics provider. If we are using it's helm chart to deploy the Cluster Agent we will have to set the following values:

datadog:
  appKey: abcd...
  apiKey: efgh...

clusterAgent:
  enabled: true
  metricsProvider:
    enabled: true

Once the Cluster Agent have been upgraded and the metrics provider is available:

$ kubectl get pods -n datadog
NAME                                          READY   STATUS    RESTARTS   AGE
datadog-2pc4x                                 3/3     Running   0          7m13s
datadog-cluster-agent-6cc7cf55d6-6946r        1/1     Running   1          120m
datadog-kube-state-metrics-699964c777-wrkwz   1/1     Running   0          100m
datadog-xk79r                                 3/3     Running   0          6m49s
$ kubectl get apiservice | grep datadog
v1beta1.external.metrics.k8s.io        datadog/datadog-cluster-agent-metrics-api   True        100m

We can now start pushing HPA objects with references to external metrics, for example:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: ampa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ampa
  minReplicas: 2
  maxReplicas: 22
  metrics:
    - type: External
      external:
        metricName: kubernetes.kubelet.cpu.usage
        metricSelector:
          matchLabels:
            cluster_name: example
        targetAverageValue: 50

The important setting that controls how the HPA behaves are:

  • metricName: Defines the metrics we are going to use to autoscale, it can be any metric that the Cluster Agent retrieves, for example kubernetes_state.deployment.replicas_available or kubernetes.kubelet.cpu.usage
  • metricsSelector.matchLabels: We can specify a set of tags we are going to use to filter the metric
  • targetValue or targetAverageValue: We are setting the desired value of the metric that will be used to scale up and down the objecy defined under spec.scaleTargetRef. If we use targetValue, it's values is the absolute value. For targetAverageValue we are defining the individual value for each replica (so it divides the value by the number of replicas)

We can achieve similar results using prometheus as an external metrics provider

Update: To use the final autoscaling/v2 version there are some minor changes:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ampa
spec:
  minReplicas: 2
  maxReplicas: 22
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: spin-fiat
  metrics:
    - type: External
      external:
      metric:
        name: kubernetes.kubelet.cpu.usage
        selector:
          matchLabels:
            cluster_name: example
      target:
        averageValue: "50"
        type: AverageValue

We can use kubectl convert to adjust the existing manifests.

Posted on 10/01/2022

Categories