Using Labels to Filter Kubernetes Namespace Query Results in Prometheus

prometheus filter namespace label join group_left kube_namespace_labels

1 min read | by Jordi Prats

We can use the labels of a namespace to narrow down the results of a query. For example. we are going to write a query to identify Kubernetes namespaces that have external secrets in a non-ready state and belong to a specific team (we are going to use team-a in this example).

The query is going to be the following:

sum(externalsecret_status_condition{condition!="Ready"}) by (name, namespace) \
* on (namespace) group_left() kube_namespace_labels{label_pet2cattle_com_team="team-a"} > 0

This first part is the unfiltered query:

sum(externalsecret_status_condition{condition!="Ready"}) by (name, namespace)

The second part if joining the results with the kube_namespace_labels metric, which contains metadata labels for Kubernetes namespaces:

on (namespace) group_left() kube_namespace_labels{label_pet2cattle_com_team="team-a"}

Since we are filtering by a specific label using {label_pet2cattle_com_team="team-a"}, this query will only consider objects belonging to a namespace with this label.

Posted on 19/04/2023

Categories