Span Profiles with Grafana Tempo and Pyroscope

The docker compose consists of:

• Ride share demo application
• Tempo
• Pyroscope
• Grafana

rideshare applications generate traces and profiling data that should be available in Grafana. Pyroscope and Tempo datasources are provisioned automatically.

Build and run

The project can be run locally with the following commands:

# Pull latest pyroscope and grafana images:
docker pull grafana/pyroscope:latest
docker pull grafana/grafana:latest

docker-compose up

Navigate to the Explore page, select a trace and click on one of its spans that have a linked profile:

By default, only the root span gets labeled (the first span created locally): such spans are marked with the link icon and have pyroscope.profile.id attribute set to the corresponding span ID. Please note that presence of the attribute does not necessarily indicate that the span has a profile: stack trace samples might not be collected, if the utilized CPU time is less than the sample interval (10ms).

Instrumentation

• rideshare demo application instrumented with OpenTelemetry:
  • Go OTel integration
  • Java OTel integration
• pyroscope itself is instrumented with OpenTelemetry and otel-profiling-go for profiling integration.

Grafana Tempo configuration

In order to correlate trace spans with profiling data, Tempo datasource should be configured:

• Data source of the profiling data.
• Tags to use in the query.
• Profile type: as of now, only CPU time profile is fully supported.
• Query override.

While tags are optional, configuring them is highly recommended for optimizing query performance. In our example, we configured the host.name tag for use in Pyroscope queries as the hostname label. This configuration restricts the data set for lookup to a specific host, ensuring that queries remain consistently fast. Note that the tags you configure must be present in the spans attributes or resources for a trace to profiles span link to appear.

Please refer to our documentation for more details.