Running tcpdump on an OpenShift cluster
3 min read | by Jordi Prats
If we want to take a look at the network traffic that we get out of an OpenShift node we can use the oc debug command to spin up a privileged pod with tcpdump installed. This way we don't need to ssh into the worker node.
First let's create a Pod that generates some traffic that we want to be able to see, for example:
$ kubectl run test --image nginx
Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/test created
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test 1/1 Running 0 10s 10.12.13.14 ip-10-12-13-14.eu-central-1.compute.internal <none> <none>
$ kubectl exec -it test -- sh
# while true; do timeout 1 bash -c 'cat < /dev/null > /dev/tcp/pet2cattle.com/80'; echo $?; sleep 1; done
0
0
(...)
With k get pods -o wide we'll be able to see on which node we'll need to run the tcpdump command (NODE column).
We can then run oc debug to spin up the debug Pod on the node:
$ oc debug -t node/ip-10-12-13-14.eu-central-1.compute.internal
Warning: would violate PodSecurity "restricted:v1.24": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/ip-10-12-13-14eu-central-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.12.13.14
If you don't see a command prompt, try pressing enter.
sh-4.4# tcpdump -nni ens5 'host 44.237.70.75'
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens5, link-type EN10MB (Ethernet), capture size 262144 bytes
05:45:44.930055 IP 10.12.13.14.47354 > 44.237.70.75.80: Flags [S], seq 3878788409, win 26583, options [mss 8861,sackOK,TS val 1277714585 ecr 0,nop,wscale 7], length 0
05:45:45.071413 IP 44.237.70.75.80 > 10.12.13.14.47354: Flags [S.], seq 2709617984, ack 3878788410, win 62293, options [mss 1460,sackOK,TS val 529986443 ecr 1277714585,nop,wscale 7], length 0
05:45:45.072131 IP 10.12.13.14.47354 > 44.237.70.75.80: Flags [.], ack 1, win 208, options [nop,nop,TS val 1277714727 ecr 529986443], length 0
05:45:45.072837 IP 10.12.13.14.47354 > 44.237.70.75.80: Flags [F.], seq 1, ack 1, win 208, options [nop,nop,TS val 1277714728 ecr 529986443], length 0
05:45:45.213010 IP 44.237.70.75.80 > 10.12.13.14.47354: Flags [F.], seq 1, ack 2, win 487, options [nop,nop,TS val 529986585 ecr 1277714728], length 0
05:45:45.213094 IP 10.12.13.14.47354 > 44.237.70.75.80: Flags [.], ack 2, win 208, options [nop,nop,TS val 1277714868 ecr 529986585], length 0
This command is going to automatically create a Pod with the right privileges, running in the specified node:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ip-10-12-13-14eu-central-1computeinternal-debug 1/1 Running 0 2s 10.12.13.14 ip-10-12-13-14.eu-central-1.compute.internal <none> <none>
Posted on 22/05/2023