Using the nodeSelector for selecting a kubernetes cluster partition

4 min read | by Jordi Prats

On a kubernetes cluster not all nodes are expected to have the same amount of resources so we might need to schedule some pods on specific nodes due to the resources they have (for example access to a GPU) or due to the network connectivity they have (for example edge nodes). Using nodeSelector we can the scheduler how we want out pods to be scheduled

On another pods we saw how to assign a pod to a specific node but, generally speaking, we won't want to select a specific node: what we would like to do is to set a group of nodes on which we would like the pod to be scheduled.

To simulate this environment we are going to using a kubernetes cluster with multiple nodes using minkube:

$ kubectl get nodes
NAME           STATUS   ROLES                  AGE     VERSION
minikube       Ready    control-plane,master   8m      v1.20.0
minikube-m02   Ready    <none>                 5m52s   v1.20.0
minikube-m03   Ready    <none>                 4m23s   v1.20.0
minikube-m04   Ready    <none>                 3m9s    v1.20.0
minikube-m04   Ready    <none>                 113s    v1.20.0

To classify the nodes we are going to use labels. We can set them using kubectl label as follows:

kubectl label nodes <nodename> <label>=<value>

For example, for setting a label called "nodegroup" setting it's value to "alpha" for node minikube-m04 and minikube-m05 we would execute the following commands:

kubectl label nodes minikube-m04 nodegroup=alpha
kubectl label nodes minikube-m05 nodegroup=alpha

To be able to use these labels to schedule a pod we will have to use a nodeSelector key on the spec section. Under this key we will have to add all the labels we want to use to schedule this pod (we can also set it on a deployment on the spec.template.spec section). For example:

apiVersion: v1
kind: Pod
metadata:
  name: demo
spec:
  nodeSelector:
    nodegroup: alpha
  containers:
    - name: demo
      image: busybox
      command: ["sleep"]
      args: ["1h"]

By deploying this manifest the scheduler will search for a node that matches the nodeSelector labels:

$ kubectl apply -f demo.yaml
pod/demo created

Using the -o wide option we will be able to see the selected node:

$ kubectl get pod demo -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
demo   1/1     Running   0          19s   10.244.3.2   minikube-m04   <none>           <none>

We can also use this manifest to create multiple pods to see where it is scheduled:

$ for i in $(seq 1 10); do cat demo.yaml | sed "s/demo/demo-$i/g" | kubectl apply -f -; done
pod/demo-1 created
pod/demo-2 created
pod/demo-3 created
(...)

We will be able to see that only the pods with the label (minikube-m04 and minikube-m05) are going to be selected regardless of any empty node on the cluster

$ kubectl get pod -o wide | grep demo
demo      1/1     Running             0          2m10s   10.244.3.2   minikube-m04   <none>           <none>
demo-1    1/1     Running             0          33s     10.244.4.2   minikube-m05   <none>           <none>
demo-10   0/1     ContainerCreating   0          23s     <none>       minikube-m05   <none>           <none>
demo-2    1/1     Running             0          32s     10.244.3.3   minikube-m04   <none>           <none>
demo-3    1/1     Running             0          32s     10.244.4.3   minikube-m05   <none>           <none>
demo-4    1/1     Running             0          30s     10.244.3.4   minikube-m04   <none>           <none>
demo-5    1/1     Running             0          29s     10.244.4.4   minikube-m05   <none>           <none>
demo-6    0/1     ContainerCreating   0          27s     <none>       minikube-m05   <none>           <none>
demo-7    0/1     ContainerCreating   0          27s     <none>       minikube-m05   <none>           <none>
demo-8    1/1     Running             0          26s     10.244.3.5   minikube-m04   <none>           <none>
demo-9    1/1     Running             0          24s     10.244.3.6   minikube-m04   <none>           <none>

Posted on 18/03/2021