docs/content/advanced/metrics.md
+++ title = "Metrics" description = "Monitor Watermill in realtime" date = 2019-02-12T21:00:00+01:00 weight = -400 draft = false bref = "Monitor Watermill in realtime using Prometheus" +++
Monitoring of Watermill may be performed by using decorators for publishers/subscribers and middlewares for handlers. We provide a default implementation using Prometheus, based on the official Prometheus client for Go.
The components/metrics package exports PrometheusMetricsBuilder, which provides convenience functions to wrap publishers, subscribers and handlers so that they update the relevant Prometheus registry:
{{% load-snippet-partial file="src-link/components/metrics/builder.go" first_line_contains="// PrometheusMetricsBuilder" last_line_contains="func (b PrometheusMetricsBuilder)" %}}
If you are using Watermill's router (which is recommended in most cases), you can use a single convenience function AddPrometheusRouterMetrics to ensure that all the handlers added to this router are wrapped to update the Prometheus registry, together with their publishers and subscribers:
{{% load-snippet-partial file="src-link/components/metrics/builder.go" first_line_contains="// AddPrometheusRouterMetrics" last_line_contains="AddMiddleware" padding_after="1" %}}
Example use of AddPrometheusRouterMetrics:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="// we leave the namespace" last_line_contains="metricsBuilder.AddPrometheusRouterMetrics" %}}
In the snippet above, we have left the namespace and subsystem arguments empty. The Prometheus client library uses these to prefix the metric names. You may want to use namespace or subsystem, but be aware that this will impact the metric names and you will have to adjust the Grafana dashboard accordingly.
The PrometheusMetricsBuilder allows for custom configuration of histogram buckets by setting the PublishBuckets or HandlerBuckets field.
If HandlerBuckets is not provided, default watermill's values will be used, which are one order of magnitude smaller than default buckets (5ms10s), because the handler execution times are typically shorter (µsms range).
For PublishBuckets, the default values are the same as the default Prometheus buckets (5ms~10s).
Standalone publishers and subscribers may also be decorated through the use of dedicated methods of PrometheusMetricBuilder:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="subWithMetrics, err := " last_line_contains="pubWithMetrics, err := " padding_after="3" %}}
In accordance with how Prometheus works, the service needs to expose a HTTP endpoint for scraping. By convention, it is a GET endpoint, and its path is usually /metrics.
To serve this endpoint, there are two convenience functions, one using a previously created Prometheus Registry, while the other also creates a new registry:
{{% load-snippet-partial file="src-link/components/metrics/http.go" first_line_contains="// CreateRegistryAndServeHTTP" last_line_contains="func ServeHTTP(" %}}
Here is an example of its use in practice:
{{% load-snippet-partial file="src-link/_examples/basic/4-metrics/main.go" first_line_contains="prometheusRegistry, closeMetricsServer :=" last_line_contains="metricsBuilder.AddPrometheusRouterMetrics" %}}
To see how the metrics dashboard works in practice, you can check out the metrics example.
Follow the instructions in the example's README to make it run and add the Prometheus data source to Grafana.
We have prepared a Grafana dashboard to use with the metrics implementation described above. It provides basic information about the throughput, failure rates and publish/handler durations.
If you want to check out the dashboard on your machine, you can use the Example application.
To find out more about the metrics that are exported to Prometheus, see Exported metrics.
<a target="_blank" href="https://threedots.tech/watermill-io/grafana_dashboard.png"></a>
To import the Grafana dashboard, select Dashboard/Manage from the left menu, and then click on +Import.
Enter the dashboard URL https://grafana.com/dashboards/9777 (or just the ID, 9777), and click on Load.
Then select your Prometheus data source that scrapes the /metrics endpoint. Click on Import, and you're done!
Listed below are all the metrics that are registered on the Prometheus Registry by PrometheusMetricsBuilder.
For more information on Prometheus metric types, please refer to Prometheus docs.
<table> <tr> <th>Object</th> <th>Metric</th> <th>Description</th> <th>Labels/Values</th> </tr> <tr> <td rowspan="3">Subscriber</td> <td rowspan="3"><code>subscriber_messages_received_total</code></td> <td rowspan="3">A Prometheus Counter. Counts the number of messages obtained by the subscriber.</td> <td><code>acked</code> is either "acked" or "nacked".</td> </tr> <tr> <td><code>handler_name</code> is set if the subscriber operates within a handler; "<no handler>" otherwise.</td> </tr> <tr> <td><code>subscriber_name</code> identifies the subscriber. If it implements <code>fmt.Stringer</code>, it is the result of `String()`, <code>package.structName</code> otherwise.</td> </tr> <tr> <td rowspan="2">Handler</td> <td rowspan="2"><code>handler_execution_time_seconds</code></td> <td rowspan="2">A Prometheus Histogram. Registers the execution time of the handler function wrapped by the middleware.</td> <td><code>handler_name</code> is the name of the handler.</td> </tr> <tr> <td><code>success</code> is either "true" or "false", depending on whether the wrapped handler function returned an error or not.</td> </tr> <tr> <td rowspan="3">Publisher</td> <td rowspan="3"><code>publish_time_seconds</code></td> <td rowspan="3">A Prometheus Histogram. Registers the time of execution of the Publish function of the decorated publisher.</td> <td><code>success</code> is either "true" or "false", depending on whether the decorated publisher returned an error or not.</td> </tr> <tr> <td><code>handler_name</code> is set if the publisher operates within a handler; "<no handler>" otherwise.</td> </tr> <tr> <td><code>publisher_name</code> identifies the publisher. If it implements <code>fmt.Stringer</code>, it is the result of `String()`, <code>package.structName</code> otherwise.</td> </tr> </table>Additionally, every metric has the node label, provided by Prometheus, with value corresponding to the instance that the metric comes from, and job, which is the job name specified in the Prometheus configuration file.
NOTE: As described above, using non-empty namespace or subsystem will result in prefixed metric names. You might need to adjust for it, for example in the definitions of panels in the Grafana dashboard.
If you feel like some metric is missing, you can easily expand this basic implementation. The best way to do so is to use the prometheus registry that is used with the ServeHTTP method and register a metric according to the documentation of the Prometheus client.
An elegant way to update these metrics would be through the use of decorators:
{{% load-snippet-partial file="src-link/message/decorator.go" first_line_contains="// MessageTransformSubscriberDecorator" last_line_contains="type messageTransformSubscriberDecorator" %}}
and/or router middlewares.
A more simplistic approach would be to just update the metric that you want in the handler function.