Using ephemeral containers
3 min read | by Jordi Prats
Starting Kubernetes 1.23, ephemeral containers are enabled by default (in beta though). Using ephemeral containers we can now troubleshoot pods by deploying a temporary container into it with extra privileges or binaries to use
If we try run commands on an existing container, with certain images we might not have all the tools we need to troubleshoot an issue, we might not even have ps or netstat:
$ ps
sh: 1: ps: not found
$ netstat
sh: 2: netstat: not found
We can circumvent some times these issues using the /proc filesystem to identify processes or connections but for certain distro-less containers we might not even have a shell:
$ kubectl exec -it coredns-d76bd69b-sr9ns -- sh
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "49c0d3d2625fea119a6148f59862b28cd2b6179f21c14ac612043507e7f2f2fe": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown
Using kubectl debug we can deploy a a temporary container (with any image) into the pod to run debugging commands, since this new container will share the process namespace with the other containers running in the pod:
$ kubectl exec -it unpriv -- id
uid=101(nginx) gid=101(nginx) groups=101(nginx)
$ kubectl debug unpriv -it --image=busybox -- id
Defaulting debug container name to debugger-7vkpp.
If you don't see a command prompt, try pressing enter.
Error attaching, falling back to logs: unable to upgrade connection: container debugger-7vkpp not found in pod unpriv_kube-system
uid=0(root) gid=0(root) groups=10(wheel)
We can also instruct kubectl debug to create a copy of the Pod instead of modifying the existing one with the --copy-to option, for example:
$ kubectl debug unpriv -it --image=alpine --copy-to=unpriv-debug
Defaulting debug container name to debugger-wqwqp.
If you don't see a command prompt, try pressing enter.
/ #
We can not check with kubectl get pods how it has created a copy of the Pod before adding the ephemeral container:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
unpriv 1/1 Running 0 2d9h
unpriv-debug 2/2 Running 0 10m
Futhermore, since within the ephemeral container we are going to have root privilages, we'll be able to install new tools without having to hack around the limitations of the existing container:
$ kubectl debug unpriv -it --image=alpine -- sh
Defaulting debug container name to debugger-hjnft.
If you don't see a command prompt, try pressing enter.
/ # apk add -u tcpdump
(1/2) Installing libpcap (1.10.1-r0)
(2/2) Installing tcpdump (4.99.1-r3)
Executing busybox-1.35.0-r17.trigger
OK: 5 MiB in 16 packages
Posted on 22/08/2022