Creating a kubernetes cluster with multiple nodes using minkube
6 min read | by Jordi Prats
Minikube is very useful for creating mockup environments for testing purposes, but it can be also use for studying yout CKA certification. Some topics, like node affinity or node failure will require you to have a multinode cluster but you can still use minikube
To start a 3 node cluster using minikube you just need to add the --node option to the start command, for example:
$ minikube start --nodes 3
😄 minikube v1.16.0 on Ubuntu 20.04
✨ Automatically selected the docker driver. Other choices: kvm2, none
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
💾 Downloading Kubernetes v1.20.0 preload ...
> preloaded-images-k8s-v8-v1....: 491.00 MiB / 491.00 MiB 100.00% 17.20 Mi
🔥 Creating docker container (CPUs=2, Memory=2633MB) ...
🐳 Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: storage-provisioner, default-storageclass
👍 Starting node minikube-m02 in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=2633MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.49.2
🐳 Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
▪ env NO_PROXY=192.168.49.2
🔎 Verifying Kubernetes components...
👍 Starting node minikube-m03 in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=2633MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.49.2,192.168.49.3
🐳 Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
▪ env NO_PROXY=192.168.49.2
▪ env NO_PROXY=192.168.49.2,192.168.49.3
🔎 Verifying Kubernetes components...
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Once it finishes you can use kubectl get nodes to check that you have a 3-node cluster:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 24m v1.20.0
minikube-m02 Ready <none> 22m v1.20.0
minikube-m03 Ready <none> 21m v1.20.0
What minikube does is to spin up three docker containers, one per node:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8eabef14b49f gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 "/usr/local/bin/entr…" 21 minutes ago Up 21 minutes 127.0.0.1:32787->22/tcp, 127.0.0.1:32786->2376/tcp, 127.0.0.1:32785->5000/tcp, 127.0.0.1:32784->8443/tcp minikube-m03
7b62f81d236a gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 "/usr/local/bin/entr…" 22 minutes ago Up 22 minutes 127.0.0.1:32783->22/tcp, 127.0.0.1:32782->2376/tcp, 127.0.0.1:32781->5000/tcp, 127.0.0.1:32780->8443/tcp minikube-m02
afd1211a1178 gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 "/usr/local/bin/entr…" 24 minutes ago Up 24 minutes 127.0.0.1:32779->22/tcp, 127.0.0.1:32778->2376/tcp, 127.0.0.1:32777->5000/tcp, 127.0.0.1:32776->8443/tcp minikube
We can easily simulate a node failure by stopping one of the worker containers. Before doing it, let's create a deployment with 6 replicas to simulate our pods:
$ kubectl create deployment demo --image=nginx --replicas=6
deployment.apps/demo created
Pods will be distributed between the two worker nodes we have:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
demo-997c454df-kj4j7 1/1 Running 0 47s 10.244.2.2 minikube-m03 <none> <none>
demo-997c454df-pzks9 1/1 Running 0 47s 10.244.1.3 minikube-m02 <none> <none>
demo-997c454df-q7br7 1/1 Running 0 47s 10.244.1.2 minikube-m02 <none> <none>
demo-997c454df-s4zrg 1/1 Running 0 47s 10.244.2.4 minikube-m03 <none> <none>
demo-997c454df-shtgs 1/1 Running 0 47s 10.244.1.4 minikube-m02 <none> <none>
demo-997c454df-v8c2q 1/1 Running 0 47s 10.244.2.3 minikube-m03 <none> <none>
Now we can simulate a crash on one of the nodes by stopping one of the worker nodes, since the docker containers have it's node name as the container name it's quite simple:
$ docker stop minikube-m03
minikube-m03
Shortly afterwards, with kubectl get nodes we will be able to see that the cluster realizes that the node it not longer in Ready state:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 4h8m v1.20.0
minikube-m02 Ready <none> 4h6m v1.20.0
minikube-m03 NotReady <none> 4h5m v1.20.0
Despite that, pods won't move immediately, we can use kubectl describe on one of the pods that where on the failed node to see the events:
$ kubectl describe pod demo-997c454df-kj4j7
Name: demo-997c454df-kj4j7
Namespace: default
Priority: 0
Node: minikube-m03/192.168.49.4
Start Time: Tue, 05 Jan 2021 16:16:14 +0100
Labels: app=demo
pod-template-hash=997c454df
Annotations: <none>
Status: Running
IP: 10.244.2.2
IPs:
IP: 10.244.2.2
Controlled By: ReplicaSet/demo-997c454df
Containers:
nginx:
Container ID: docker://48f40340c742d6aeb03613633108c1ef3791bea219f769288a8f1f825483b981
Image: nginx
Image ID: docker-pullable://nginx@sha256:4cf620a5c81390ee209398ecc18e5fb9dd0f5155cd82adcbae532fec94006fb9
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 05 Jan 2021 16:16:44 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-lscdz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
default-token-lscdz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-lscdz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m58s default-scheduler Successfully assigned default/demo-997c454df-kj4j7 to minikube-m03
Normal Pulling 5m53s kubelet Pulling image "nginx"
Normal Pulled 5m34s kubelet Successfully pulled image "nginx" in 18.580908133s
Normal Created 5m28s kubelet Created container nginx
Normal Started 5m26s kubelet Started container nginx
Warning DNSConfigForming 2m52s (x6 over 5m58s) kubelet Search Line limits were exceeded, some search paths have been omitted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local int.compumark.com dresources.com markmonitor.com
Warning NodeNotReady 91s node-controller Node is not ready
Once we can see the NodeNotReady on the pod's events, it won't try to recover the lost pods for 5 minutes (it is configurable via the --pod-eviction-timeout flag on the controller manager) After 5 minutes it will terminate the dead pods, and recreate new pods on the other node to be able to achieve the deployment's desired state:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
demo-997c454df-d8ng4 1/1 Running 0 27s 10.244.1.7 minikube-m02 <none> <none>
demo-997c454df-kj4j7 1/1 Terminating 0 9m59s 10.244.2.2 minikube-m03 <none> <none>
demo-997c454df-ksk9d 1/1 Running 0 27s 10.244.1.5 minikube-m02 <none> <none>
demo-997c454df-pzks9 1/1 Running 0 9m59s 10.244.1.3 minikube-m02 <none> <none>
demo-997c454df-q7br7 1/1 Running 0 9m59s 10.244.1.2 minikube-m02 <none> <none>
demo-997c454df-s4zrg 1/1 Terminating 0 9m59s 10.244.2.4 minikube-m03 <none> <none>
demo-997c454df-shtgs 1/1 Running 0 9m59s 10.244.1.4 minikube-m02 <none> <none>
demo-997c454df-v8c2q 1/1 Terminating 0 9m59s 10.244.2.3 minikube-m03 <none> <none>
demo-997c454df-wf5p7 1/1 Running 0 27s 10.244.1.6 minikube-m02 <none> <none>
Posted on 19/01/2021