Installing and configuring AWS Karpenter

7 min read | by Jordi Prats

Starting November 29th 2021, AWS is considering that Karpenter is ready for production: It is a cluster autoscaler alternative intended to improve the efficiency and cost of running workloads on Kubernetes clusters

Karpenter's key inner workings are these two control loops:

  • A fast-acting allocator that makes sure that pods are scheduled as quickly as possible
  • A slow-acting reallocator that replaces nodes and consolidates the workload

To define what worker nodes it can be spawn, we can configure Provisioners with a set of requirements that constrain what nodes can be provisioned. Karpenter will choose the best-fitting node to spin up depending on the Pods that are in Pending state using the fast-acting control loop.

As workloads get in and out of the cluster, the slow-acting control loop will make sure it's workloads are not fragmented across multiple nodes and will try to consolidate them into as few nodes as possible.

Karpenter installation

To install and use Karpenter, first we will need to create an IRSA enabled role with the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateLaunchTemplate",
        "ec2:CreateFleet",
        "ec2:RunInstances",
        "ec2:CreateTags",
        "iam:PassRole",
        "ec2:TerminateInstances",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeInstances",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceTypeOfferings",
        "ec2:DescribeAvailabilityZones",
        "ssm:GetParameter"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Once this policy is attached to the IRSA IAM role that we are going to use with Karpenter's ServiceAccount we can set the annotation on the values.yaml like this:

serviceAccount:
  create: true
  name: karpenter
  annotations:
    "eks.amazonaws.com/role-arn": arn:...

On this file we also need to configure the controller. We will need to set the AWS_REGION, the clusterName and the clusterEndpoint.

To get the clusterEndpoint we can use aws cli to retrieve the URL using it's clusterName like this:

$ aws eks describe-cluster --name pet2cattle --query "cluster.endpoint" --output json
"https://B2BC91B51F0003EA14AADED1D2FFBB1C.gr7.eu-west-1.eks.amazonaws.com"

Once we have all the data we can push the configuration to the values.yaml file:

controller:
  env:
  - name: AWS_REGION
    value: eu-west-1
  clusterName: "pet2cattle"
  clusterEndpoint: "..."

Once we have this config in place we can deploy it using Karpenter's helm chart (at the time of this writing it's version is 0.5.0)

$ helm repo add karpenter https://charts.karpenter.sh
$ helm repo update
$ helm upgrade --install karpenter karpenter/karpenter -n karpenter -f values.yaml

Once deployed we'll have to wait until it's available:

$ kubectl get pods -n karpenter
NAME                                    READY   STATUS    RESTARTS   AGE
karpenter-controller-6fdec9addf-qwert   1/1     Running   0          11m
karpenter-webhook-dffdeb86ad-pl111      1/1     Running   0          10m

At this point we are now ready to configure Karpenter to deploy new nodes to the cluster, so we will need to make sure Cluster Autoscaler is disabled so they don't compete adding nodes to the cluster.

To configure Karpenter to create new nodes we will need to create a Provisioner that will be used to determine which node needs to be added. We can configure the following:

  • spec.ttlSecondsUntilExpired: It sets when a node expires, meaning that we can configure nodes to get refreshed after a given amount of time
  • spec.ttlSecondsAfterEmpty: Sets the amount of time it waits before deleting a node due to low utilization
  • spec.taints: List of taints to add to the node
  • spec.labels: Labels to configured to the node
  • spec.limits: Defines a limit the scaling that Karpenter will perform in order to control customer costs. This would prevent scheduling more nodes when, for a set of circumstances, the scheduled nodes may not be able to satisfy the pending pods and the this could potentially lead to a situation when Karpenter spins new nodes indefinitely.
  • spec.requirements: Contains the requirements that will constrain the parameters of provisioned nodes, we can for example, choose whether we want to use spot instances, the architecture of the nodes or the instance-type
  • spec.provider: Contains the provider's configuration, for example the tags we want to set, it's intance profile and other selectors like the securityGroupSelector

A Provisioner object will look like follows:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: pet2cattle-workers
spec:
  ttlSecondsUntilExpired: 2592000

  ttlSecondsAfterEmpty: 30

  labels:
    nodelabel: example

  requirements:
    - key: "node.kubernetes.io/instance-type" 
      operator: In
      values: ["m5a.large", "m5a.xlarge", "m5a.2xlarge"]
    - key: "topology.kubernetes.io/zone" 
      operator: In
      values: ["es-west-1a", "eu-west-1b", "eu-west-1c"]
    - key: "kubernetes.io/arch" 
      operator: In
      values: ["arm64", "amd64"]
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["spot", "on-demand"]

  provider:
    instanceProfile: 'eks_pet2cattle_worker-instance-profile'
    securityGroupSelector:
      Name: 'eks_pet2cattle-worker'
    tags:
      exampleTag: TagValue

  limits:
    resources:
      cpu: 1000

At this point we just need to push it into Kubernetes:

$ kubectl apply -f karpenter-provisioner.yaml

If we look at Karpenter's logs we will be able to see how it spins up new nodes:

$ stern karpenter -n karpenter
(...)
karpenter-controller-6fdec9addf-qwert manager 2021-11-29T22:58:44.238Z  INFO  controller.provisioning Starting provisioner  {"commit": "84b683b", "provisioner": "pet2cattle-workers"}
karpenter-controller-6fdec9addf-qwert manager 2021-11-29T22:58:44.239Z  INFO  controller.provisioning Waiting for unschedulable pods  {"commit": "84b683b", "provisioner": "pet2cattle-workers"}
karpenter-controller-6fdec9addf-qwert manager 2021-11-29T22:58:48.516Z  INFO  controller.provisioning Batched 1 pods in 1.000595223s  {"commit": "84b683b", "provisioner": "pet2cattle-workers"}
karpenter-controller-6fdec9addf-qwert manager 2021-11-29T22:58:48.523Z  INFO  controller.provisioning Computed packing of 1 node(s) for 1 pod(s) with instance type option(s) [m5a.large m5a.xlarge m5a.2xlarge]  {"commit": "84b683b", "provisioner": "pet2cattle-workers"}
karpenter-controller-6fdec9addf-qwert manager 2021-11-29T22:58:50.748Z  INFO  controller.provisioning Launched instance: i-d8d1ea0b8ede8e690, hostname: ip-10-12-15-228.eu-west-1.compute.internal, type: m5a.large, zone: eu-west-1a, capacityType: spot {"commit": "84b683b", "provisioner": "pet2cattle-workers"}
karpenter-webhook-dffdeb86ad-pl111 webhook 2021-11-29T22:58:50.764Z INFO  webhook Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0xc000680900), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, */*"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"6243"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*http.body)(0xc00084e680), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6243, TransferEncoding:[]string(nil), Close:false, Host:"karpenter-webhook.karpenter.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"10.12.15.143:49608", RequestURI:"/default-resource?timeout=10s", TLS:(*tls.ConnectionState)(0xc0004b1d90), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc00084e6c0)}  {"commit": "84b683b"}
karpenter-webhook-dffdeb86ad-pl111 webhook 2021-11-29T22:58:50.765Z INFO  webhook Kind: "karpenter.sh/v1alpha5, Kind=Provisioner" PatchBytes: null  {"commit": "84b683b", "knative.dev/kind": "karpenter.sh/v1alpha5, Kind=Provisioner", "knative.dev/namespace": "", "knative.dev/name": "pet2cattle-workers", "knative.dev/operation": "UPDATE", "knative.dev/resource": "karpenter.sh/v1alpha5, Resource=provisioners", "knative.dev/subresource": "status", "knative.dev/userinfo": "{system:serviceaccount:karpenter:karpenter 4cc8c7b5-cc9b-48a1-8862-c41b97416ab2 [system:serviceaccounts system:serviceaccounts:karpenter system:authenticated] map[authentication.kubernetes.io/pod-name:[karpenter-controller-6fdec9addf-qwert] authentication.kubernetes.io/pod-uid:[acfe89ab-dead-beef-beef-caaad8320d0f]]}"}
karpenter-webhook-dffdeb86ad-pl111 webhook 2021-11-29T22:58:50.765Z INFO  webhook remote admission controller audit annotations=map[string]string(nil)  {"commit": "84b683b", "knative.dev/kind": "karpenter.sh/v1alpha5, Kind=Provisioner", "knative.dev/namespace": "", "knative.dev/name": "pet2cattle-workers", "knative.dev/operation": "UPDATE", "knative.dev/resource": "karpenter.sh/v1alpha5, Resource=provisioners", "knative.dev/subresource": "status", "knative.dev/userinfo": "{system:serviceaccount:karpenter:karpenter 4cc8c7b5-cc9b-48a1-8862-c41b97416ab2 [system:serviceaccounts system:serviceaccounts:karpenter system:authenticated] map[authentication.kubernetes.io/pod-name:[karpenter-controller-6fdec9addf-qwert] authentication.kubernetes.io/pod-uid:[acfe89ab-dead-beef-beef-caaad8320d0f]]}", "admissionreview/uid": "891234ff-beef-dead-adde-f6d760b8babc", "admissionreview/allowed": true, "admissionreview/result": "nil"}
karpenter-controller-6fdec9addf-qwert manager 2021-11-29T22:58:50.769Z  INFO  controller.provisioning Bound 1 pod(s) to node ip-10-12-15-228.eu-west-1.compute.internal {"commit": "84b683b", "provisioner": "pet2cattle-workers"}
karpenter-webhook-dffdeb86ad-pl111 webhook 2021-11-29T22:58:50.772Z INFO  webhook Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0xc000681c20), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, */*"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"6243"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*http.body)(0xc00084fd40), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6243, TransferEncoding:[]string(nil), Close:false, Host:"karpenter-webhook.karpenter.svc:443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"10.12.15.143:49610", RequestURI:"/validate-resource?timeout=10s", TLS:(*tls.ConnectionState)(0xc0000b0bb0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc00084fd80)} {"commit": "84b683b"}
karpenter-webhook-dffdeb86ad-pl111 webhook 2021-11-29T22:58:50.774Z INFO  webhook remote admission controller audit annotations=map[string]string(nil)  {"commit": "84b683b", "knative.dev/kind": "karpenter.sh/v1alpha5, Kind=Provisioner", "knative.dev/namespace": "", "knative.dev/name": "pet2cattle-workers", "knative.dev/operation": "UPDATE", "knative.dev/resource": "karpenter.sh/v1alpha5, Resource=provisioners", "knative.dev/subresource": "status", "knative.dev/userinfo": "{system:serviceaccount:karpenter:karpenter 4cc8c7b5-cc9b-48a1-8862-c41b97416ab2 [system:serviceaccounts system:serviceaccounts:karpenter system:authenticated] map[authentication.kubernetes.io/pod-name:[karpenter-controller-6fdec9addf-qwert] authentication.kubernetes.io/pod-uid:[acfe89ab-dead-beef-beef-caaad8320d0f]]}", "admissionreview/uid": "cea2eecd-7c0b-48c2-9a92-c79e4e1476f7", "admissionreview/allowed": true, "admissionreview/result": "nil"}
karpenter-controller-6fdec9addf-qwert manager 2021-11-29T22:58:50.778Z  INFO  controller.provisioning Waiting for unschedulable pods  {"commit": "84b683b", "provisioner": "pet2cattle-workers"}

We'll need to wait for a while for the node to become ready:

$ kubectl get nodes
NAME                                         STATUS     ROLES    AGE   VERSION
ip-10-12-14-223.eu-west-1.compute.internal   Ready      <none>   8d    v1.21.4-eks-033ce7e
ip-10-12-14-131.eu-west-1.compute.internal   Ready      <none>   38d   v1.21.4-eks-033ce7e
ip-10-12-15-123.eu-west-1.compute.internal   Ready      <none>   49m   v1.21.4-eks-033ce7e
ip-10-12-15-36.eu-west-1.compute.internal    NotReady   <none>   24s   
ip-10-12-17-87.eu-west-1.compute.internal    Ready      <none>   9h    v1.21.4-eks-033ce7e
$ kubectl get nodes
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-12-14-223.eu-west-1.compute.internal   Ready    <none>   8d      v1.21.4-eks-033ce7e
ip-10-12-14-131.eu-west-1.compute.internal   Ready    <none>   38d     v1.21.4-eks-033ce7e
ip-10-12-15-123.eu-west-1.compute.internal   Ready    <none>   51m     v1.21.4-eks-033ce7e
ip-10-12-15-36.eu-west-1.compute.internal    Ready    <none>   2m54s   v1.21.5-eks-bc4871b
ip-10-12-17-87.eu-west-1.compute.internal    Ready    <none>   9h      v1.21.4-eks-033ce7e

If we kubectl describe the node we will be able to see that the label we have defined on the Provisioner object:

$ kubectl describe node ip-10-12-15-36.eu-west-1.compute.internal
Name:               ip-10-12-15-36.eu-west-1.compute.internal
Roles:              <none>
Labels:             nodelabel=example
                    karpenter.sh/capacity-type=spot
                    karpenter.sh/provisioner-name=pet2cattle-workers
                    node.kubernetes.io/instance-type=m5a.large
                    topology.kubernetes.io/zone=eu-west-1a
Annotations:        node.alpha.kubernetes.io/ttl: 0
CreationTimestamp:  Thu, 01 Dec 2021 01:34:36 +0100
Taints:             node.kubernetes.io/unreachable:NoExecute
                    karpenter.sh/not-ready:NoSchedule
                    node.kubernetes.io/unreachable:NoSchedule
(...)

For spot instances: To be able to properly manage it's lifecycle, we will have to make sure the termination-handler works together with Karpenter by draining the node when a termination notice is received.


Posted on 03/12/2021