Kubernetes: nginx ingress controller - failed calling webhook

3 min read

On a kubernetes cluster you might find the following error:

$ kubectl apply -f ingress.yaml 
Error from server (InternalError): error when creating "ingress": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s: service "ingress-nginx-controller-admission" not found

This message is telling you that the nginx ingress controller is not responding: This can be due to the fact that it might have partially uninstalled it (for example, by removing the namespace) but some of the global objects still present (for example the ValidatingWebhookConfiguration object).

To remove the ValidatingWebhookConfiguration object for the nginx ingress controller is named ingress-nginx-admission, we can remove it with kubectl delete like this:

$ kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted

Once you have it removed, you will be able to create ingress objects as usual:

$ kubectl apply -f ingress.yaml
ingress.extensions/testingress created

We should be considering this a workaround, we need to keep in mind that removing a namespace is not equivalent of uninstalling an application: Some applications might have created objects that are not namespaced:

$ kubectl api-resources --namespaced=false
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
componentstatuses                 cs           v1                                     false        ComponentStatus
namespaces                        ns           v1                                     false        Namespace
nodes                             no           v1                                     false        Node
persistentvolumes                 pv           v1                                     false        PersistentVolume
mutatingwebhookconfigurations                  admissionregistration.k8s.io/v1        false        MutatingWebhookConfiguration
validatingwebhookconfigurations                admissionregistration.k8s.io/v1        false        ValidatingWebhookConfiguration
customresourcedefinitions         crd,crds     apiextensions.k8s.io/v1                false        CustomResourceDefinition
apiservices                                    apiregistration.k8s.io/v1              false        APIService
tokenreviews                                   authentication.k8s.io/v1               false        TokenReview
selfsubjectaccessreviews                       authorization.k8s.io/v1                false        SelfSubjectAccessReview
selfsubjectrulesreviews                        authorization.k8s.io/v1                false        SelfSubjectRulesReview
subjectaccessreviews                           authorization.k8s.io/v1                false        SubjectAccessReview
certificatesigningrequests        csr          certificates.k8s.io/v1                 false        CertificateSigningRequest
eniconfigs                                     crd.k8s.amazonaws.com/v1alpha1         false        ENIConfig
ingressclassparams                             elbv2.k8s.aws/v1beta1                  false        IngressClassParams
flowschemas                                    flowcontrol.apiserver.k8s.io/v1beta1   false        FlowSchema
prioritylevelconfigurations                    flowcontrol.apiserver.k8s.io/v1beta1   false        PriorityLevelConfiguration
nodes                                          metrics.k8s.io/v1beta1                 false        NodeMetrics
ingressclasses                                 networking.k8s.io/v1                   false        IngressClass
runtimeclasses                                 node.k8s.io/v1                         false        RuntimeClass
podsecuritypolicies               psp          policy/v1beta1                         false        PodSecurityPolicy
clusterrolebindings                            rbac.authorization.k8s.io/v1           false        ClusterRoleBinding
clusterroles                                   rbac.authorization.k8s.io/v1           false        ClusterRole
priorityclasses                   pc           scheduling.k8s.io/v1                   false        PriorityClass
csidrivers                                     storage.k8s.io/v1                      false        CSIDriver
csinodes                                       storage.k8s.io/v1                      false        CSINode
storageclasses                    sc           storage.k8s.io/v1                      false        StorageClass
volumeattachments                              storage.k8s.io/v1                      false        VolumeAttachment

Thus, be aware that by removing the nginx ingress controller namespace and the ingress-nginx-admission webhook you are not completely removed all the objects that were installed with the nginx ingress controller: You also want to check for the presence of some clusterroles and clusterrolebindings to also get rid of them:

$ kubectl get clusterrole | grep nginx
ingress-nginx                                                          2021-01-26T19:23:38Z
ingress-nginx-admission                                                2021-01-26T19:23:47Z
$ kubectl get clusterrolebinding | grep nginx
ingress-nginx                                          ClusterRole/ingress-nginx                                          7d20h
ingress-nginx-admission                                ClusterRole/ingress-nginx-admission                                7d20h

Posted on 26/02/2021