Use Grafana As The UI

SkyWalking provides PromQL Service, LogQL Service and TraceQL Service. You can choose Grafana as the SkyWalking UI. About the installation and how to use please refer to the official document.

Notice <1>, Gafana is AGPL-3.0 license, which is very different from Apache 2.0. Please follow AGPL 3.0 license requirements.

Notice <2>, SkyWalking always uses its native UI as first class. All visualization features are only available on native UI. Grafana UI is an extension on our support of PromQL APIs. We don't maintain or promise the complete Grafana UI dashboard setup.

Configure Data Source

Prometheus Data Source

In the data source config panel, choose the Prometheus and set the url to the OAP server address, the default port is 9090.

SkyWalking Data Source

Before you start, please install the SkyWalking data source plugin. In the data source config panel, choose the SkyWalking and set the url to the OAP server graphql service address, the default port is 12800.

Loki Data Source

In the data source config panel, choose the Loki and set the url to the OAP server address, the default port is 3100.

Tempo Data Source

In the data source config panel, choose the Tempo and set the url to the full OAP trace API address, including the context path. The default port is 3200, for example http://<oap-host>:3200/skywalking for SkyWalking native trace or http://<oap-host>:3200/zipkin for Zipkin trace. The SkyWalking native trace API context path is /skywalking, and the Zipkin trace API context path is /zipkin. You can customize them via SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING and SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN respectively. If you customize either path, use the customized path in the Grafana Tempo data source URL.

Notice: The feature requires version Grafana 12 or later. And require disabling the Streaming option for the Tempo data source in the following configuration.

SkyWalking Native Trace

Zipkin Trace

Configure Metric Dashboards

Dashboards Settings

The following steps are the example of config a General Service dashboard:

1. Create a dashboard named General Service. A layer is recommended as a dashboard.
2. Configure variables for the dashboard:

After configure, you can select the service/instance/endpoint on the top of the dashboard:

Add Panels

The following contents show how to add several typical metrics panels. General settings:

1. Chose the metrics and chart.
2. Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
3. Add PromQL expressions, use the variables configured above for the labels then you can select the labels value from top. Note: Some metrics values may be required calculations to match units.
4. Select the returned labels you want to show on panel.
5. Test query and save the panel.

Common Value Metrics

1. For example service_apdex and Time series chart.
2. Add PromQL expression, the metric scope is Service, so add labels service and layer for match.
3. Set Connect null values --> Always and Show points --> Always because when the query interval > 1hour or 1day SkyWalking return the hour/day step metrics values.

Labeled Value Metrics

1. For example service_percentile and Time series chart.
2. Add PromQL expressions, the metric scope is Service, add labels service and layer for match. And it's a labeled value metric, add labels='0,1,2,3,4' filter the result label, and addrelabels='P50,P75,P90,P95,P99' rename the result label.
3. Set Connect null values --> Always and Show points --> Always because when the query interval > 1hour or 1day SkyWalking return the hour/day step metrics values.

Sort Metrics

1. For example service_instance_cpm and Bar gauge chart.
2. Add PromQL expressions, add labels parent_service and layer for match, add top_n='10' and order='DES' filter the result.
3. Set the Calculation --> Latest*.

Sampled Records

Same as the Sort Metrics.

Configure Topology Dashboards

Dashboards Settings

For now, SkyWalking support General Service and Service Mesh topology dashboards, the layer is GENERAL and MESH respectively. The following configuration can reuse the above General Service dashboard and add a new variable Plugin_SkyWalking for the dashboard:

Add Topology Panel

1. Chose the Node Graph chart.
2. Set Layer and Service by the variables. If you want to show all services in this layer, set Service empty.
3. Set Node Metrics and Edge Metrics which you want to show on the topology.

Configure Log Dashboard

Dashboards Settings

The following steps are the example of config a log dashboard:

1. Create a dashboard named Log.

2. Configure variables for the dashboard:

3. Please make sure service_instance and endpoint variable enabled Include All option and set Custom all value to * or blank (typed by space button on the keyboard):

4. Tags variable is a little different from others, for more details, please refer Ad hoc filters:

5. After configure, you can select log query variables on the top of the dashboard:

Add Log Panel

The following steps show how to add a log panel.

1. Choose Logs chart.
2. Set the Line limit value (The max number of logs to return in a query) and Order value (Determines the sort order of logs).
3. Add LogQL expressions, use the variables configured above for the labels and searching keyword. service_instance & endpoint variable ref should use raw variable-format-options to prevent it value be escaped.
4. Test query and save the panel.

Configure Relation Dashboards

Dashboards Settings

The following steps are the example of config a General Relation dashboard:

1. Create a dashboard named General Relation.

2. Configure variables for the dashboard:

   After configure, you can select the service/instance/endpoint and the dest service/instance/endpoint on the top of the dashboard:

Add Relation Metrics Panel

The following contents show how to add relation metric panels.

1. Chose the metrics and chart.
2. Set Query options, Connect null values, Show points and Returned Labels. Please refer Configure Metric Dashboards part.
3. Add PromQL expression, For all scopes, set Layer, Dest Layer, Service and Dest Service by the variables. For service instance relation scope, extra set Service Instance and Dest Service Instance. For endpoint relation scope, extra set Endpoint and Dest Endpoint.
4. Test query and save the panel.

Configure Trace Dashboard

SkyWalking Native Trace

Dashboards Settings

The following steps are an example of configuring a Trace dashboard for SkyWalking native trace:

1. Go to the Explore page, select the Tempo data source named SkyWalkingTraceQL (the name you configured). Add it to the dashboard and save it as SkyWalking Trace.

2. Configure variables for the dashboard: Notice: The variables (Service/Instance/Endpoint) querying leverage the Prometheus data source SkyWalkingPromQL(DS_SW_Prometheus).

3. If you want the Service/Instance/Endpoint variables to support the All option, enable Allow custom values and set Custom all value to * so it matches the backend wildcard.

Add Trace Panel

The following steps show how to add a trace panel for SkyWalking native trace.

1. Choose Table chart or edit your saved panel.

2. Set Query type to Search and set the query condition to variables, such as Service Name to $Service.

3. Set other query conditions if needed.

4. Test query and save the panel.

5. If you click the trace ID on the trace panel, it will jump to the explore page and show the trace details.

Zipkin Trace

Same as the SkyWalking native trace, but use the Zipkin Tempo data source you configured. The following are the variables and trace panel configuration screenshots for Zipkin trace:

Preview on demo.skywalking.a.o

SkyWalking community provides a preview site for services of General and Service Mesh layers from the demo environment. You could take a glance through Preview metrics on Grafana of the demo deployment.

Notice, we don't provide all setups due to our monitoring target expanding fast. This demo is for helping you understand the above documents only.