Monitoring APIRequestCount in OpenShift

OpenShift APIRequestCount monitoring API usage

2 min read | by Jordi Prats

Openshift provides an object that tracks the number of requests made to the Kubernetes API server. It provides insights into the load on the cluster, the performance of applications, and helps in capacity planning. By monitoring APIRequestCount, you can identify potential bottlenecks, detect unusual spikes in traffic, and optimize resource allocation.

$ kubectl get apirequestcounts
NAME                                                                           REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
alertmanagerconfigs.v1alpha1.monitoring.coreos.com                                                6                       1706
alertmanagers.v1.monitoring.coreos.com                                                            20                      2891
apiservices.v1.apiregistration.k8s.io                                                             994                     99521
(...)

We can get the details for each of the available APIs. For example, we can use the following command to get stats about the calls issued to Pod. It will fetch the APIRequestCount data for Pods from the Kubernetes API server and display the results. The output will include information such as the API group, resource (Pods), version, namespace, and the number of requests made for each specific Pod:

$ kubectl get apirequestcounts pods.v1
NAME      REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
pods.v1                      2161                    137560

We can get an even more detailed response by retrieving the whole object using -o yaml:

$ kubectl get apirequestcounts pods.v1 -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-07-08T19:56:30Z"
  generation: 1
  name: pods.v1
  resourceVersion: "108979491"
  uid: 041cb88b-9d93-a411-dead-72a5bf79fa94
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - byUser:
      - byVerb:
        - requestCount: 87
          verb: get
        - requestCount: 21
          verb: list
        - requestCount: 2
          verb: watch
        requestCount: 110
        userAgent: Go-http-client/2.0
        username: system:serviceaccount:openshift-etcd-operator:etcd-operator
      - byVerb:
        - requestCount: 109
          verb: watch
        requestCount: 109
        userAgent: Prometheus/2.32.1
        username: system:serviceaccount:openshift-monitoring:prometheus-k8s
      - byVerb:
        - requestCount: 34
          verb: get
        requestCount: 34
        userAgent: Mozilla/5.0
        username: cluster-admin
      - byVerb:
        - requestCount: 26
          verb: get
        - requestCount: 6
          verb: update
        requestCount: 32
        userAgent: multus/v0.0.0
        username: system:serviceaccount:openshift-multus:multus
      - byVerb:
        - requestCount: 7
          verb: create
        - requestCount: 14
          verb: delete
        - requestCount: 6
          verb: watch
        requestCount: 27
        userAgent: catalog/v0.0.0
        username: system:serviceaccount:openshift-operator-lifecycle-manager:olm-operator-serviceaccount

By regularly monitoring APIRequestCount, setting up alerts, and analyzing the data over time, you can ensure the smooth operation of your applications and make informed decisions about resource allocation

Posted on 11/07/2023

Categories