Controller Manager Stats

Version history

• December 2022, v1.25.x

Resources

• Process metrics
• Rest client metrics
• Node collector metrics
• Workqueue metrics
• Metrics general information

Metrics insight

Metrics used are either stable (not explicit) or alpha (explicit).

• process_cpu_seconds_total

• process_resident_memory_bytes

• process_virtual_memory_bytes

• process_open_fds

• process_start_time_seconds

• process_max_fds

• rest_client_requests_total (alpha)

  • code
  • host
  • method

• rest_client_response_size_bytes (alpha)

  • host
  • verb

• rest_client_request_size_bytes (alpha)

  • host
  • verb

• rest_client_request_duration_seconds (alpha)

  • host
  • verb

• workqueue_longest_running_processor_seconds (alpha)

  • name

• workqueue_unfinished_work_seconds (alpha)

  • name

• workqueue_adds_total (alpha)

  • name

• workqueue_depth (alpha)

  • name

• workqueue_retries_total (alpha)

  • name

• workqueue_work_duration_seconds (alpha)

  • name

• node_collector_evictions_total

  • zone

• node_collector_unhealthy_nodes_in_zone (alpha)

  • zone

• node_collector_zone_size (alpha)

  • zone

• leader_election_master_status (alpha)

  • name

Setup environment for manual tests

WIP: controller manager will usually run at every master node, but that might not be the case. It could be executed as a host process or an in-cluster pod.

• If host process (for example, systemd), metricbeat should be running at that same node gathering data from the controller.
• If executing as a pod:
  • A metricbeat instance can be also executed using the same affinity and deployment object (deployment, daemonset, ...) as the controller manager.

Generating expectation files

In order to support a new Kubernetes release you'll have to generate new expectation files for this module in _meta/test. For that, start by deploying a new kubernetes cluster on the required Kubernetes version, for example:

kind create cluster --image kindest/node:v1.32.0

After that, you can apply the kubernetes.yml file from the root of the kubernetes module:

kubectl apply -f kubernetes.yml

This is required since accessing the controllermanager metrics requires additional permissions.

Then you can expose the controllermanager api:

kubectl -n kube-system port-forward pod/kube-controller-manager-kind-control-plane 10257

Save the metrics output from https://localhost:10257/metrics to a new _meta/test/metrics.x.xx file.

curl -k https://localhost:10257/metrics > _meta/test/metrics.x.xx

Run the following commands to generate and test the expected files:

cd metricbeat/module/kubernetes/controllermanager
# generate the expected files
go test ./... --data
# test the expected files
go test ./...