Collecting Metrics with Prometheus

Envoy Metrics

Envoy typically exposes metrics through an endpoint on its admin interface. To avoid exposing the entire admin interface to Prometheus (and other workloads in the cluster), Contour configures a static listener that sends traffic to the stats endpoint and nowhere else.

Envoy supports Prometheus-compatible /stats/prometheus endpoint for metrics on port 8002.

Contour Metrics

Contour exposes a Prometheus-compatible /metrics endpoint that defaults to listening on port 8000. This can be configured by using the --http-address and --http-port flags for the serve command.

Note: the Service deployment manifest when installing Contour must be updated to represent the same port as the configured flag.

The metrics endpoint exposes the following metrics:

{{% metrics-table %}}

Deploy Sample Monitoring Stack

Follow the instructions here to install a monitoring stack to your cluster using the kube-prometheus project sample manifests. These instructions install the Prometheus Operator, a Prometheus instance, a Grafana Deployment, and other components. Note that this is a quickstart installation, see documentation here for more details on customizing the installation for production usage.

The instructions above show how to access the Prometheus and Grafana web interfaces using kubectl port-forward. Sample HTTPProxy resources in the examples/ directory can also be used to access these through your Contour installation:

$ kubectl apply -f examples/prometheus/httpproxy.yaml
$ kubectl apply -f examples/grafana/httpproxy.yaml

Scrape Contour and Envoy metrics

To enable Prometheus to scrape metrics from the Contour and Envoy pods, we can add some RBAC customizations with a Role and RoleBinding in the projectcontour namespace:

kubectl apply -f examples/prometheus/rbac.yaml

Now add PodMonitor resources for scraping metrics from Contour and Envoy pods in the projectcontour namespace:

kubectl apply -f examples/prometheus/podmonitors.yaml

You should now be able to browse Contour and Envoy Prometheus metrics in the Prometheus and Grafana web interfaces to create dashboards and alerts.

Apply Contour and Envoy Grafana Dashboards

Some sample Grafana dashboards are provided as ConfigMap resources in the examples/grafana directory. To use them with your Grafana installation, apply the resources:

$ kubectl apply -f examples/grafana/dashboards.yaml

And update the Grafana Deployment:

$ kubectl -n monitoring patch deployment grafana --type=json --patch-file examples/grafana/deployment-patch.json

You should now see dashboards for Contour and Envoy metrics available in the Grafana web interface.