Troubleshoot Azure Monitor data source issues

This document provides solutions to common issues you may encounter when configuring or using the Azure Monitor data source, organized by the stage where the issue occurs.

Connection and network errors

These errors indicate Grafana can't reach Azure endpoints. Verify connectivity before investigating authentication or query issues.

"Connection refused" or timeout errors

Symptoms:

• Data source test fails with network errors
• Queries timeout without returning results

Solutions:

1. Verify network connectivity from Grafana to Azure endpoints.
2. Check firewall rules allow outbound HTTPS (port 443) to Azure services.
3. For private networks, ensure Private Link or VPN is configured correctly.
4. For Grafana Cloud, configure Private Data Source Connect if accessing private resources.

SSL/TLS certificate errors

Symptoms:

• Certificate validation failures
• SSL handshake errors

Solutions:

1. Ensure the system time is correct (certificate validation fails with incorrect time).
2. Verify corporate proxy isn't intercepting HTTPS traffic.
3. Check that required CA certificates are installed on the Grafana server.

Authentication errors

These errors occur when credentials configured in the data source or in Azure are invalid, expired, or missing permissions.

"Authorization failed" or "Access denied"

Symptoms:

• Save & test fails with "Authorization failed"
• Queries return "Access denied" errors
• Subscriptions don't load when clicking Load Subscriptions

Possible causes and solutions:

Cause	Solution
App registration doesn't have required permissions	Assign the Reader role to the app registration on the subscription or resource group you want to monitor. Refer to the Azure documentation for role assignments.
Incorrect tenant ID, client ID, or client secret	Verify the credentials in the Azure Portal under App registrations > your app > Overview (for IDs) and Certificates & secrets (for secret).
Client secret has expired	Create a new client secret in Azure and update the data source configuration.
Managed Identity not enabled on the Azure resource	For VMs, enable managed identity in the Azure Portal under Identity. For App Service, enable it under Identity in the app settings.
Managed Identity not assigned the Reader role	Assign the Reader role to the managed identity on the target subscription or resources.

"Invalid client secret" or "Client secret not found"

Symptoms:

• Authentication fails immediately after configuration
• Error message references invalid credentials

Solutions:

1. Ensure you copied the client secret value, not the secret ID. In Azure Portal under Certificates & secrets, the secret value is only shown once when created. The secret ID is a different identifier and won't work for authentication.
2. Verify the client secret was copied correctly (no extra spaces or truncation).
3. Check if the secret has expired in Azure Portal under App registrations > your app > Certificates & secrets.
4. Create a new secret and update the data source configuration.

"Tenant not found" or "Invalid tenant ID"

Symptoms:

• Data source test fails with tenant-related errors
• Unable to authenticate

Solutions:

1. Verify the Directory (tenant) ID in Azure Portal under Microsoft Entra ID > Overview.
2. Ensure you're using the correct Azure cloud setting (Azure, Azure Government, or Azure China).
3. Check that the tenant ID is written using a valid GUID format.

Client certificate authentication not working

Symptoms:

• Authentication fails when using App Registration with client certificate
• Error references invalid certificate or signature validation failure

Possible causes and solutions:

Cause	Solution
Certificate has expired	Check the certificate expiration date in Azure Portal under App registrations > your app > Certificates & secrets. Upload a new certificate if expired.
Wrong certificate format	Verify you're using the correct format (PEM or PFX) and that the content matches what you selected. PEM requires separate certificate and private key fields.
PFX password is incorrect	For encrypted PFX certificates, verify the password is correct.
Private key doesn't match the certificate	Ensure the private key corresponds to the certificate uploaded to Azure. Regenerate the certificate and key pair if needed.
Certificate not uploaded to Azure	Verify the public certificate is uploaded to the app registration under Certificates & secrets > Certificates in the Azure Portal.

Grafana server configuration errors

These errors occur when the Grafana server .ini configuration is missing settings required by Managed Identity, Workload Identity, or Current User authentication. If your Azure credentials are correct but authentication still fails, check this section.

Managed Identity not working

Symptoms:

• Managed Identity option is available but authentication fails
• Error: "Managed identity authentication is not available"

Solutions:

1. Verify managed_identity_enabled = true is set in the Grafana server configuration under [azure].
2. Confirm the Azure resource hosting Grafana has managed identity enabled.
3. For user-assigned managed identity, ensure managed_identity_client_id is set correctly.
4. Verify the managed identity has the Reader role on the target resources.
5. Restart Grafana after changing server configuration.

Workload Identity not working

Symptoms:

• Workload Identity authentication fails in Kubernetes/AKS environment
• Token file errors

Solutions:

1. Verify workload_identity_enabled = true is set in the Grafana server configuration.
2. Check that the service account is correctly annotated for workload identity.
3. Verify the federated credential is configured in Azure.
4. Ensure the token path is accessible to the Grafana Pod.
5. Check the workload identity webhook is running in the cluster.

"401 Unauthorized" despite correct Azure setup

Symptoms:

• Authentication fails with 401 Unauthorized even though Managed Identity or Workload Identity is correctly configured in Azure
• The managed identity has the correct RBAC roles
• managed_identity_enabled or workload_identity_enabled is set to true in the Grafana .ini file

Cause: The forward_settings_to_plugins setting under [azure] doesn't include the Azure Monitor plugin, so the plugin doesn't receive the Azure authentication settings from the Grafana server.

Solutions:

1. Check if you've customized the forward_settings_to_plugins setting under [azure] in your Grafana .ini file.
2. If customized, ensure grafana-azure-monitor-datasource is listed. For example:
   iniCopy

   [azure]
   managed_identity_enabled = true
   forward_settings_to_plugins = grafana-azure-monitor-datasource
   

3. By default, Grafana includes all Grafana Labs Azure plugins. If you haven't customized this setting, the default includes the Azure Monitor plugin.
4. Restart Grafana after making changes.

Query errors

These errors occur when executing queries against Azure Monitor services.

"No data" or empty results

Symptoms:

• Query executes without error but returns no data
• Charts show "No data" message

Possible causes and solutions:

Cause	Solution
Time range doesn't contain data	Expand the dashboard time range or verify data exists in Azure Portal.
Wrong resource selected	Verify you've selected the correct subscription, resource group, and resource.
Metric not available for resource	Not all metrics are available for all resources. Check available metrics in Azure Portal under the resource's Metrics blade.
Metric has no values	Some metrics only populate under certain conditions (for example, error counts when errors occur).
Permissions issue	Verify the identity has read access to the specific resource.

"Bad request" or "Invalid query"

Symptoms:

• Query fails with 400 error
• Error message indicates query syntax issues

Solutions for Logs queries:

1. Validate your KQL syntax in the Azure Portal Log Analytics query editor.
2. Check for typos in table names or column names.
3. Ensure referenced tables exist in the selected workspace.
4. Verify the time range is valid (not in the future, not too far in the past for data retention).

Solutions for Metrics queries:

1. Verify the metric name is valid for the selected resource type.
2. Check that dimension filters use valid dimension names and values.
3. Ensure the aggregation type is supported for the selected metric.

"Resource not found"

Symptoms:

• Query fails with 404 error
• Resource picker shows resources that can't be queried

Solutions:

1. Verify the resource still exists in Azure (it may have been deleted or moved).
2. Check that the subscription is correct.
3. Refresh the resource picker by re-selecting the subscription.
4. Verify the identity has access to the resource's resource group.

Logs query timeout

Symptoms:

• Query runs for a long time then fails with DatasourceError or context deadline exceeded
• Alert evaluations fail with timeout errors
• Error mentions timeout or query limits

Cause: The default data source timeout in Grafana is 30 seconds, but Azure Log Analytics queries can take up to 3 minutes on the Azure side, especially for complex KQL queries or large time ranges.

Solutions:

1. Increase the data source timeout. Open the Azure Monitor data source settings, scroll to the Misc section, and increase the Timeout to 300 (seconds). Refer to Query timeout for details.
2. Narrow the time range to reduce data volume.
3. Add filters to reduce the result set.
4. Use summarize to aggregate data instead of returning raw rows.
5. Consider using Basic Logs for large datasets (if enabled).
6. Break complex queries into smaller parts.
7. For alerting, ensure the alert evaluation interval is long enough to accommodate the query duration plus the configured timeout.

"Metrics not available" for a resource

Symptoms:

• Resource appears in picker but no metrics are listed
• Metric dropdown is empty

Solutions:

1. Verify the resource type supports Azure Monitor metrics.
2. Check if the resource is in a region that supports metrics.
3. Some resources require diagnostic settings to emit metrics—configure these in Azure Portal.
4. Try selecting a different namespace for the resource.

Azure Resource Graph errors

These errors are specific to Azure Resource Graph (ARG) queries.

"Query execution failed"

Symptoms:

• ARG query fails with execution errors
• Results don't match expected resources

Solutions:

1. Validate query syntax in Azure Portal Resource Graph Explorer.
2. Check that you have access to the subscriptions being queried.
3. Verify table names are correct (for example, Resources, ResourceContainers).
4. Some ARG features require specific permissions, check ARG documentation.

Query returns incomplete results

Symptoms:

• Not all expected resources appear in results
• Results seem truncated

Solutions:

1. ARG queries are paginated. The data source handles pagination automatically, but very large result sets may be limited.
2. Add filters to reduce result set size.
3. Verify you have access to all subscriptions containing the resources.

Application Insights Traces errors

These errors are specific to the Traces query type.

"No traces found"

Symptoms:

• Trace query returns empty results
• Operation ID search finds nothing

Solutions:

1. Verify the Application Insights resource is collecting trace data.
2. Check that the time range includes when the traces were generated.
3. Ensure the Operation ID is correct (copy directly from another trace or log).
4. Verify the identity has access to the Application Insights resource.

Basic Logs errors

These errors are specific to Basic Logs queries.

Basic Logs toggle is not available

Symptoms:

• The Analytics / Basic toggle doesn't appear in the Logs query editor
• Unable to switch to Basic Logs mode

Solutions:

1. Verify Enable Basic Logs is toggled on in the data source configuration. Refer to Enable Basic Logs.
2. Basic Logs isn't available when creating alert rules. If you're in the alerting UI, use Analytics mode instead.

Basic Logs query returns errors

Symptoms:

• Query fails with "BadArgumentError" or similar errors
• Query returns unexpected results

Solutions:

1. Ensure you've selected only a single workspace resource. Basic Logs doesn't support multi-resource queries.
2. Remove any time-range filters from the query itself. Basic Logs always uses the dashboard time range.
3. Verify the table you're querying is configured with the Basic or Auxiliary log plan in Azure. Not all tables support Basic Logs.
4. Check that your KQL query uses only supported operators.

Unexpected costs from Basic Logs

Solutions:

1. Basic Logs queries are billed per query by Azure, separate from Grafana costs. Review the Azure pricing documentation.
2. Check your query frequency. Dashboard auto-refresh and variable changes can trigger repeated queries.
3. Consider switching infrequently queried tables back to Analytics mode to avoid per-query charges.

Alerting errors

These errors occur when using Azure Monitor data in Grafana alert rules or recording rules. For general alerting configuration, refer to the Azure Monitor alerting documentation.

Alert failure with Current User authentication

Symptoms:

• Alert rules and recording rules fail with authentication errors
• Dashboard queries work, but alerting evaluations do not
• Error messages reference missing or invalid credentials during background evaluation

Cause: Current User authentication relies on the logged-in user's credentials, which aren't available for background operations like alerting.

Solutions:

1. Configure fallback service credentials in the data source settings. Typically this means adding an App Registration (client secret) as the fallback.
2. Verify the fallback credentials have the required Azure RBAC permissions on the resources used in your alert rules.
3. Refer to Limitations and fallback credentials for configuration details.

Alert evaluation timeout

Symptoms:

• Alert rules fail with DatasourceError or context deadline exceeded
• Alerts work intermittently, failing when queries take longer than usual

Solutions:

1. Increase the data source timeout. The default is 30 seconds, but Azure Log Analytics queries can take up to 3 minutes. Open the Azure Monitor data source settings, scroll to Misc, and set Timeout to 300. Refer to Query timeout.
2. Ensure the alert evaluation interval is long enough to accommodate the query duration.
3. Simplify complex KQL queries used in alert rules.
4. Reduce the time range in Log Analytics queries.

Template variable errors

For general template variable configuration, refer to the template variables documentation.

"Properties found in series but missing valueProp and textProp"

Symptoms:

• Template variables using query types such as Resource Groups, Resource Names, or Subscriptions fail with this error
• Variables that previously worked stop returning values after a Grafana upgrade
• Dashboards that rely on cascading variables break

Cause: This error occurs when the Azure Monitor plugin's variable query response format doesn't match what Grafana expects. This was a known bug that affected specific Grafana versions.

Solutions:

1. Check your Grafana version. This issue was a software bug fixed in later releases. Update to the latest Grafana patch release for your version.
2. Grafana Cloud users: If you're on a release channel that doesn't yet include the fix, switch to the stable channel in your Cloud stack settings (Stack Management > General > Release channel) to receive the latest stable patch.
3. Self-hosted users: Download the latest patch release from the Grafana download page.
4. After upgrading, reload the dashboard and re-run the variable queries.

Variables return no values

Solutions:

1. Verify the data source connection is working (test it in the data source settings).
2. Check that parent variables (for cascading variables) have valid selections.
3. Verify the identity has permissions to list the requested resources.
4. For Logs variables, ensure the KQL query returns a single column.

Variables stop working after a Grafana upgrade

Symptoms:

• Template variables that worked before an upgrade return errors or empty results
• Dashboard variable dropdowns show no options or display error indicators

Solutions:

1. Check the Grafana release notes for breaking changes related to template variables or the Azure Monitor data source.
2. Verify the Azure Monitor plugin version is compatible with your Grafana version. Go to Administration > Plugins and data > Plugins, search for "Azure Monitor", and check the version.
3. Clear the browser cache and reload the dashboard.
4. Re-open the variable configuration and click Run query to verify it returns expected results.
5. If the issue persists, check for known bugs in the Grafana GitHub issues and consider rolling back to the previous Grafana version.

Variables are slow to load

Solutions:

1. Set variable refresh to On dashboard load instead of On time range change.
2. Reduce the scope of variable queries (for example, filter by resource group instead of entire subscription).
3. For Logs variables, optimize the KQL query to return results faster.

Enable debug logging

To capture detailed error information for troubleshooting Azure Monitor issues:

1. Set the Grafana log level to debug in the configuration file:

   iniCopy

   [log]
   level = debug
   

2. Restart Grafana for the change to take effect.

3. Reproduce the issue and review the logs at /var/log/grafana/grafana.log (or your configured log location).

4. Look for entries containing azuremonitor or azure for Azure Monitor-specific request and response details.

5. Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you've tried the solutions in this document and still encounter issues:

1. Check the Grafana community forums for similar issues.
2. Review the Azure Monitor data source GitHub issues for known bugs.
3. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user.
4. When reporting issues, include:
   • Grafana version
   • Error messages (redact sensitive information)
   • Steps to reproduce
   • Relevant configuration (redact credentials)