docs/sources/datasources/tempo/troubleshooting/index.md
This document provides solutions to common issues you may encounter when configuring or using the Tempo data source in Grafana.
This guide covers issues related to connecting Grafana to Tempo and using the data source features. It applies to the following setups:
Where troubleshooting steps differ between these setups, the guide calls it out. Sections labeled Grafana Cloud only or self-managed Tempo apply only to those environments.
For issues with Tempo itself (not the data source), refer to the Tempo product documentation:
Additional resources for Grafana Cloud:
These errors occur when Grafana cannot establish or maintain a connection to the Tempo instance.
Error message: Failed to connect to Tempo or dial tcp: connection refused
Cause: Grafana can't reach the Tempo instance. This typically means the Tempo process isn't running, the URL in the data source settings is wrong, or a network-level rule is blocking the connection.
Solution:
3200. Common URL mistakes include:
/api or /tempo, to self-managed Tempo URLs. The correct format is http://<HOST>:3200 with no path. The /tempo suffix is only used for Grafana Cloud Traces URLs.9095) instead of the HTTP port (3200). The Tempo data source URL must be the HTTP endpoint. Grafana derives the gRPC connection for streaming from the same URL.Error message: Connection timed out or context deadline exceeded
Cause: The connection to Tempo timed out before receiving a response. This can happen when network latency is high, Tempo is overloaded, or an intermediary device (load balancer, proxy) terminates the connection before Tempo responds.
Solution:
Error message: TLS handshake failed or x509: certificate signed by unknown authority
Cause: There's a mismatch between the TLS settings in Grafana and what the Tempo instance supports or requires. For example, Tempo may present a self-signed certificate that the Grafana server doesn't trust, or the certificate's hostname doesn't match the URL.
Solution:
For additional information on setting up TLS encryption with Tempo, refer to Configure TLS communication.
These errors occur when there are issues with authentication credentials or permissions.
Error message: 401 Unauthorized or 403 Forbidden
Cause: The authentication credentials are invalid or the user doesn't have permission to access Tempo. This error appears when you select Save & test or when running a query.
Solution:
traces:read scope. Refer to Create a Cloud Access Policy token for setup instructions. To find your stack URL and credentials, refer to Locate your stack URL, user, and password.server block or reverse proxy.Error message: Tempo echo endpoint returned status 404 when selecting Save & test
Cause: The Grafana health check reaches the server but the api/echo endpoint returns a 404. The most common cause is incorrect path segments in the URL (such as /api or /tempo on a self-managed instance) that shift the health check request to a nonexistent path. A 404 can also occur when a reverse proxy in front of Tempo rejects requests that include unexpected basic authentication headers.
Solution:
http://<HOST>:3200 with no path. Refer to the Connection section for the correct format.Error message: Unauthorized when using multi-tenant Tempo
Cause: The tenant ID (X-Scope-OrgID header) is missing or incorrect. This header must be set on both the write path (in your Alloy or OpenTelemetry Collector configuration) and the read path (in the Grafana data source settings) to route data to the correct tenant.
Solution:
X-Scope-OrgID header is configured under Authentication > HTTP Headers. Refer to Multi-tenancy for data source configuration steps.X-Scope-OrgID header is set in your Alloy or OpenTelemetry Collector configuration so traces are ingested under the correct tenant.For more information, refer to Enable multi-tenancy and Tenant IDs.
These errors occur when there are issues with TraceQL queries or trace lookups.
Error message: Trace not found or trace ID not found
Cause: The specified trace ID doesn't exist in Tempo or has aged out of the configured retention period. Common reasons include:
Solution:
Verify the trace ID is correct and complete.
Check your sampling configuration in Alloy or OpenTelemetry Collector. If head or tail sampling is enabled, the trace may have been intentionally dropped. Refer to Sampling strategies for guidance.
Check that the trace is within the configured retention period for Tempo.
If using time range restrictions, expand the time range in the TraceID query settings.
Enable Use time range in query in the TraceID query settings and adjust the time shift values to search a broader range.
Verify the trace was successfully ingested by Tempo. You can run a broad query to check for recent traces:
{ } | count() > 0
If this returns results, traces are being ingested but your specific trace ID may have been dropped by sampling or aged out. For more query examples, refer to the TraceQL cookbook.
For Grafana Cloud users, refer to Troubleshoot TraceQL and search for TraceQL queries that can help investigate missing traces.
Error message: parse error or unexpected token
Cause: The TraceQL query contains syntax errors.
Solution:
span.http.status_code or resource.service.name.=, !=, =~, !~, >, <, >=, <=.Cause: The query is valid but doesn't match any traces.
Solution:
Cause: Save & test succeeds, but running queries in Explore returns no traces. The data source connection is valid, but no trace data is reaching Tempo, or the queries don't match any ingested data.
Solution:
X-Scope-OrgID header is the same on both the write path (collector) and the read path (Grafana data source). A mismatch causes traces to be ingested under one tenant and queried under another.{}, with a short, recent time range to confirm that any traces exist in Tempo. If this returns results, your original query filters are too narrow.Error message: context deadline exceeded or query timeout
Cause: The query took too long to execute. Trace searches scan raw trace data across the selected time range, so broad queries over long periods can exceed timeout limits.
Solution:
{ } | rate()) or pre-computed metrics from the metrics generator, which are optimized for aggregate workloads.Error message: localblocks processor not found or metrics-generator not configured
Cause: TraceQL metrics queries (such as { } | rate() or { } | count_over_time()) require the local-blocks processor to be active in the Tempo metrics-generator configuration. This processor is separate from the service-graphs and span-metrics processors and must be enabled independently.
Solution:
For self-managed Tempo, activate the local-blocks processor in the overrides configuration. The configuration path depends on your Tempo version and deployment method:
overrides.metrics_generator_processors: ["local-blocks"]overrides.defaults.metrics_generator.processors: [local-blocks]Refer to Configure TraceQL metrics for setup steps and required configuration.
For Grafana Cloud, the local-blocks processor is enabled by default. If you still see this error, contact Grafana Support.
Verify that the metrics generator is running and healthy by checking the Tempo metrics-generator logs for errors.
Streaming displays TraceQL query results as they become available. Without streaming, you don't see results until the query completes. Streaming is available for Grafana Cloud users by default. For self-managed Tempo, search streaming requires Tempo v2.2 or later, and metrics streaming requires Tempo v2.7 or later. Both require stream_over_http_enabled: true in the Tempo configuration.
For more information, refer to Streaming.
Cause: Results don't appear incrementally and only display after the full query completes.
Solution:
stream_over_http_enabled: true). Refer to Tempo gRPC API in the Tempo documentation. Streaming is available for Grafana Cloud users by default.Error message: rpc error: code = Unavailable desc = credentials require transport level security
Cause: Grafana tries to use gRPC over an insecure (non-TLS) connection. Streaming uses gRPC, which may require TLS depending on your Tempo configuration.
Solution:
Error message: stream error or incomplete results during streaming
Cause: The streaming connection was interrupted before all results were delivered.
Solution:
The Service Graph visualizes service dependencies and highlights request rate, error rate, and duration (RED metrics) across connections. It requires a linked Prometheus data source with service graph metrics generated by the Tempo metrics generator or Grafana Alloy. For more information, refer to Service Graph and Service Graph view.
Cause: The Service Graph view shows no data or an empty graph.
Solution:
Verify that a Prometheus data source is linked in the Tempo data source Service Graph settings under Additional settings.
For self-managed Tempo, check that the metrics generator is configured to produce service graph metrics. Refer to Service graph metrics in the Tempo documentation.
Ensure the following service graph metrics exist in Prometheus:
traces_service_graph_request_totaltraces_service_graph_request_failed_totaltraces_service_graph_request_server_seconds_sumVerify that Grafana Alloy or the Tempo metrics generator is configured to produce these metrics. These are distinct from span metrics (traces_spanmetrics_*), which power the Service Graph view table.
To verify the metrics exist, run the following query in Explore with your Prometheus data source selected:
{__name__=~"traces_service_graph_request.*"}
Check the Prometheus data source connection is working.
For Grafana Cloud Traces users, refer to Troubleshoot service graph and RED metrics.
Cause: The Service Graph node graph renders correctly, but the RED metrics table alongside it shows no data.
Solution:
traces_spanmetrics_calls_total and traces_spanmetrics_latency_bucket. Confirm that these metrics exist in your linked Prometheus data source. Newer Tempo versions may produce traces_spanmetrics_duration_seconds_bucket instead of traces_spanmetrics_latency_bucket. If you only have the newer metric name, the duration column shows no data until the Grafana frontend supports the renamed metric.Cause: Service graph or span metrics queries produce high-cardinality warnings or slow queries.
Solution:
Cause: Some services or connections are missing from the graph.
Solution:
These issues relate to the correlation features that link traces to other telemetry signals. For configuration details, refer to Trace to logs, Trace to metrics, and Trace to profiles.
Cause: Links to logs don't appear when viewing a trace.
Solution:
traceID, trace_id, or traceId). Refer to Trace to logs for common regular expression patterns by instrumentation format.Cause: The reverse direction—navigating from a Loki log line to a trace—isn't working. No trace link appears on log lines.
Solution:
Cause: Links to metrics don't appear when viewing a trace.
Solution:
$__tags variable is correctly placed in custom queries.Cause: Links to profiling data don't appear when viewing a trace.
Solution:
process_cpu:cpu:nanoseconds:cpu:nanoseconds). A mismatch between the configured profile type and the available data causes the link to not appear.Cause: The generated query doesn't match any data in the target data source.
Solution:
service.name) but Loki labels use underscores (for example, service_name). Ensure tag mappings account for this difference. Refer to Trace to logs for tag mapping configuration.pod is stored as Loki structured metadata rather than an indexed label, the auto-generated stream selector {pod="..."} returns no results. Enable Use custom query and move the tag to a pipeline filter. Refer to Trace to logs for a custom query example.These issues relate to slow queries or high resource usage.
Cause: Queries take a long time to return results.
Solution:
Cause: Large traces or result sets consume excessive memory.
Solution:
The following issues don't produce specific error messages but are commonly encountered.
Cause: The node graph visualization doesn't appear above the trace view.
Solution:
traceID and spanID, and non-root spans must include parentSpanID to establish hierarchy. If these identifiers are missing, the trace data is incomplete at the instrumentation level and the graph can't be rendered.Cause: The TraceQL editor doesn't show suggestions for attributes or values. Autocomplete requires Tempo to enumerate tags, which requires a supported storage backend (vParquet or later).
Solution:
vParquet format or later for tag-based search and autocomplete. Refer to Parquet configuration for details. Grafana Cloud Traces supports autocomplete by default.Cause: You configured settings in the Grafana UI (such as trace-to-logs links, service graph, or span time shifts), but they disappear after Grafana restarts. This happens when the data source is provisioned via a YAML configuration file, Helm chart, or Terraform. Provisioned data sources are read-only in the UI—any changes you make through the UI aren't persisted and silently revert on the next restart.
You can tell a data source is provisioned if the settings form is read-only and the button reads Test instead of Save & test.
Solution:
Cause: You can select the Tempo data source in the Grafana Alerting rule builder, but standard TraceQL search queries return trace data (spans and trace IDs), not numeric time series. Alert rules require numeric data to evaluate thresholds, so trace search queries don't produce usable alert conditions.
Solution:
TraceQL metrics queries (such as { } | rate() or { status = error } | count_over_time()) return numeric time series and can drive alert thresholds directly. However, TraceQL alerting requires the experimental tempoAlerting feature flag, which is disabled by default. To enable it, set the tempoAlerting feature flag in your Grafana configuration. For Grafana Cloud, contact Support to enable it. For step-by-step setup, refer to Trace-based alerts.
{{< admonition type="caution" >}}
The tempoAlerting feature flag is experimental. For production alerting, consider the Prometheus-based approach described below.
{{< /admonition >}}
Alternatively, use the Tempo metrics generator to produce Prometheus metrics from trace data, then create alert rules against the Prometheus data source that stores those metrics.
service-graphs, span-metrics, or local-blocks processor) to produce metrics from traces and remote-write them to a Prometheus-compatible backend.traces_spanmetrics_calls_total with a status_code="STATUS_CODE_ERROR" filter.Cause: You have multiple Tempo data sources configured (for example, one per environment or per tenant), but there's no built-in UI dropdown in Explore or dashboards to switch between them dynamically.
Workaround:
tempo type. This adds a dropdown that lets viewers switch between Tempo instances without editing the dashboard.X-Scope-OrgID header instead of separate data sources. Refer to Multi-tenancy for configuration details.These issues relate to connecting to a private Tempo instance using Private data source connect (PDC). PDC is only available in Grafana Cloud.
{{< admonition type="note" >}} Private data source connect allows Grafana Cloud to securely connect to data sources in your private network without exposing them to the public internet. For setup instructions, refer to Private data source connect. {{< /admonition >}}
Error message: No healthy PDC agents available or Private data source connect agent not connected
Cause: The PDC agent installed in your private network isn't running or can't connect to Grafana Cloud.
Solution:
For more information, refer to Private data source connect.
Error message: Failed to connect through private data source or Connection refused via PDC
Cause: The PDC agent is connected to Grafana Cloud but cannot reach the Tempo instance in your private network.
Solution:
3200) is open and accessible.curl or telnet.Error message: Request timed out via private data source connect
Cause: Queries through the PDC tunnel are timing out.
Solution:
Error message: Unauthorized or 403 Forbidden when using PDC
Cause: The authentication configured in the data source doesn't work with the PDC connection.
Solution:
X-Scope-OrgID for multi-tenant Tempo) are properly forwarded through PDC.If you continue to experience issues after following this troubleshooting guide:
When reporting issues, include: