docs/how-to/how-to-set-prometheus-in-k8s.md
This document describes how to run kata-monitor in a Kubernetes cluster using Prometheus's service discovery to scrape metrics from kata-agent.
Warning: This how-to is only for evaluation purpose, you SHOULD NOT running it in production using this configurations.
If you are running Kata containers in a Kubernetes cluster, the best way to run kata-monitor is using Kubernetes native DaemonSet, kata-monitor will run on desired Kubernetes nodes without other operations when new nodes joined the cluster.
Prometheus also support a Kubernetes service discovery that can find scrape targets dynamically without explicitly setting kata-monitor's metric endpoints.
You must have a running Kubernetes cluster first. If not, install a Kubernetes cluster first.
Also you should ensure that kubectl working correctly.
Note: More information about Kubernetes integrations:
Start Prometheus by utilizing our sample manifest:
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/docs/how-to/data/prometheus.yml
This will create a new namespace, prometheus, and create the following resources:
ClusterRole, ServiceAccount, ClusterRoleBinding to let Prometheus to access Kubernetes API server.ConfigMap that contains minimum configurations to let Prometheus run Kubernetes service discovery.Deployment that run Prometheus in Pod.Service with type of NodePort(30909 in this how to), that we can access Prometheus through <hostIP>:30909. In production environment, this type may be LoadBalancer or Ingress resource.After the Prometheus server is running, run curl -s http://hostIP:NodePort:30909/metrics, if Prometheus is working correctly, you will get response like these:
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.9403e-05
go_gc_duration_seconds{quantile="0.25"} 0.000169907
go_gc_duration_seconds{quantile="0.5"} 0.000207421
go_gc_duration_seconds{quantile="0.75"} 0.000229911
kata-monitorkata-monitor can be started on the cluster as follows:
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/docs/how-to/data/kata-monitor-daemonset.yml
This will create a new namespace kata-system and a daemonset in it.
Once the daemonset is running, Prometheus should discover kata-monitor as a target. You can open http://<hostIP>:30909/service-discovery and find kubernetes-pods under the Service Discovery list
Run this command to run Grafana in Kubernetes:
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/docs/how-to/data/grafana.yml
This will create deployment and service for Grafana under namespace prometheus.
After the Grafana deployment is ready, you can open http://hostIP:NodePort:30000/ to access Grafana server. For Grafana 7.0.5, the default user/password is admin/admin. You can modify the default account and adjust other security settings by editing the Grafana configuration.
To use Grafana show data from Prometheus, you must create a Prometheus datasource and dashboard.
datasourceOpen http://hostIP:NodePort:30000/datasources/new in your browser, select Prometheus from time series databases list.
Normally you only need to set URL to http://hostIP:NodePort:30909 to let it work, and leave the name as Prometheus as default.
A sample dashboard for Kata Containers metrics is provided which can be imported to Grafana for evaluation.
You can import this dashboard using Grafana UI, or using curl command in console.
$ curl -XPOST -i localhost:3000/api/dashboards/import \
-u admin:admin \
-H "Content-Type: application/json" \
-d "{\"dashboard\":$(curl -sL https://raw.githubusercontent.com/kata-containers/kata-containers/main/docs/how-to/data/dashboard.json )}"