Troubleshoot Kubernetes service not being resolved

When you create a Kubernetes Service, pods from within the same namespace should be able to resolve it's IP by name. For example, if we create a service named ampa-votacions; any pod from the same namespace should be able to resolve it's IP. But sometimes it can't be resolved:

$ kubectl exec -it ampa-install-ws7cw -- sh
/ $ nslookup ampa-votacions
Server:   172.20.0.10
Address:  172.20.0.10:53

** server can't find ampa-votacions.us-west-2.compute.internal: NXDOMAIN

** server can't find ampa-votacions.ampa.svc.cluster.local: NXDOMAIN

** server can't find ampa-votacions.svc.cluster.local: NXDOMAIN

** server can't find ampa-votacions.ampa.svc.cluster.local: NXDOMAIN

** server can't find ampa-votacions.cluster.local: NXDOMAIN

** server can't find ampa-votacions.svc.cluster.local: NXDOMAIN

** server can't find ampa-votacions.cluster.local: NXDOMAIN

** server can't find ampa-votacions.us-west-2.compute.internal: NXDOMAIN

One common issue is that there are no endpoints available, we can check it using kubectl describe:

$ kubectl describe svc ampa-votacions
Name:              ampa-votacions
Namespace:         ampa
Labels:            app.kubernetes.io/instance=ampa
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ampa
                   app.kubernetes.io/version=1.19.4
                   component=votacions
                   helm.sh/chart=ampa-0.1.0
Annotations:       meta.helm.sh/release-name: ampa
                   meta.helm.sh/release-namespace: ampa
Selector:          app=ampa,component=votacions
Type:              ClusterIP
IP Families:       <none>
IP:                None
IPs:               <none>
Port:              votacions  8080/TCP
TargetPort:        8080/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

We should be checking:

  • What's going on with the pods / deployments what are supposed to be behind this service
  • Check that the pods / deployments are exposing the service
  • Check that the selectorLabels are matching the correct pods

Once we find the issue, with kubectl describe svc will be able to see the presence of endpoints:

$ kubectl describe svc ampa-votacions
Name:              ampa-votacions
Namespace:         ampa
Labels:            app.kubernetes.io/instance=ampa
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ampa
                   app.kubernetes.io/version=1.19.14
                   component=votacions
                   helm.sh/chart=ampa-0.1.0
Annotations:       meta.helm.sh/release-name: ampa
                   meta.helm.sh/release-namespace: ampa
Selector:          app=ampa,component=votacions
Type:              ClusterIP
IP Families:       <none>
IP:                None
IPs:               <none>
Port:              votacions  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.103.196.193:8080
Session Affinity:  None
Events:            <none>

Once we have endpoints available, we will be able to start resolving the service within the pods


Posted on 19/04/2021