Troubleshoot issues with the Elasticsearch data source

This document provides troubleshooting information for common errors you may encounter when using the Elasticsearch data source in Grafana.

Connection errors

The following errors occur when Grafana cannot establish or maintain a connection to Elasticsearch.

Failed to connect to Elasticsearch

Error message: "Health check failed: Failed to connect to Elasticsearch"

Cause: Grafana cannot establish a network connection to the Elasticsearch server.

Solution:

1. Verify that the Elasticsearch URL is correct in the data source configuration.
2. Check that Elasticsearch is running and accessible from the Grafana server.
3. Ensure there are no firewall rules blocking the connection.
4. If using a proxy, verify the proxy settings are correct.
5. For Grafana Cloud, ensure you have configured Private data source connect if your Elasticsearch instance is not publicly accessible.

Request timed out

Error message: "Health check failed: Elasticsearch data source is not healthy. Request timed out"

Cause: The connection to Elasticsearch timed out before receiving a response.

Solution:

1. Check the network latency between Grafana and Elasticsearch.
2. Verify that Elasticsearch is not overloaded or experiencing performance issues.
3. Increase the timeout setting in the data source configuration if needed.
4. Check if any network devices (load balancers, proxies) are timing out the connection.

Failed to parse data source URL

Error message: "Failed to parse data source URL"

Cause: The URL entered in the data source configuration is not valid.

Solution:

1. Verify the URL format is correct (for example, http://localhost:9200 or https://elasticsearch.example.com:9200).
2. Ensure the URL includes the protocol (http:// or https://).
3. Remove any trailing slashes or invalid characters from the URL.

Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

Unauthorized (401)

Error message: "Health check failed: Elasticsearch data source is not healthy. Status: 401 Unauthorized"

Cause: The authentication credentials are invalid or missing.

Solution:

1. Verify that the username and password are correct.
2. If using an API key, ensure the key is valid and has not expired.
3. Check that the authentication method selected matches your Elasticsearch configuration.
4. Verify the user has the required permissions to access the Elasticsearch cluster.

Forbidden (403)

Error message: "Health check failed: Elasticsearch data source is not healthy. Status: 403 Forbidden"

Cause: The authenticated user does not have permission to access the requested resource.

Solution:

1. Verify the user has read access to the specified index.
2. Check Elasticsearch security settings and role mappings.
3. Ensure the user has permission to access the _cluster/health endpoint.
4. If using AWS Elasticsearch Service with SigV4 authentication, verify the IAM policy grants the required permissions.

Cluster health errors

The following errors occur when the Elasticsearch cluster is unhealthy or unavailable.

Cluster status is red

Error message: "Health check failed: Elasticsearch data source is not healthy"

Cause: The Elasticsearch cluster health status is red, indicating one or more primary shards are not allocated.

Solution:

1. Check the Elasticsearch cluster health using GET /_cluster/health.
2. Review Elasticsearch logs for errors.
3. Verify all nodes in the cluster are running and connected.
4. Check for unassigned shards using GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason.
5. Consider increasing the cluster's resources or reducing the number of shards.

Bad Gateway (502)

Error message: "Health check failed: Elasticsearch data source is not healthy. Status: 502 Bad Gateway"

Cause: A proxy or load balancer between Grafana and Elasticsearch returned an error.

Solution:

1. Check the health of any proxies or load balancers in the connection path.
2. Verify Elasticsearch is running and accepting connections.
3. Review proxy/load balancer logs for more details.
4. Ensure the proxy timeout is configured appropriately for Elasticsearch requests.

Index errors

The following errors occur when there are issues with the configured index or index pattern.

Index not found

Error message: "Error validating index: index_not_found"

Cause: The specified index or index pattern does not match any existing indices.

Solution:

1. Verify the index name or pattern in the data source configuration.
2. Check that the index exists using GET /_cat/indices.
3. If using a time-based index pattern (for example, [logs-]YYYY.MM.DD), ensure indices exist for the selected time range.
4. Verify the user has permission to access the index.

Time field not found

Error message: "Could not find time field '@timestamp' with type date in index"

Cause: The specified time field does not exist in the index or is not of type date.

Solution:

1. Verify the time field name in the data source configuration matches the field in your index.
2. Check the field mapping using GET /<index>/_mapping.
3. Ensure the time field is mapped as a date type, not text or keyword.
4. If the field name is different (for example, timestamp instead of @timestamp), update the data source configuration.

Query errors

The following errors occur when there are issues with query syntax or configuration.

Too many buckets

Error message: "Trying to create too many buckets. Must be less than or equal to: [65536]."

Cause: The query is generating more aggregation buckets than Elasticsearch allows.

Solution:

1. Reduce the time range of your query.
2. Increase the date histogram interval (for example, change from 10s to 1m).
3. Add filters to reduce the number of documents being aggregated.
4. Increase the search.max_buckets setting in Elasticsearch (requires cluster admin access).

Required field missing

Error message: "Required one of fields [field, script], but none were specified."

Cause: A metric aggregation (such as Average, Sum, or Min) was added without specifying a field.

Solution:

1. Select a field for the metric aggregation in the query editor.
2. Ensure the selected field exists in your index and contains numeric data.

Unsupported interval

Error message: "unsupported interval '<interval>'"

Cause: The interval specified for the index pattern is not valid.

Solution:

1. Use a supported interval: Hourly, Daily, Weekly, Monthly, or Yearly.
2. If you don't need a time-based index pattern, use No pattern and specify the exact index name.

Version errors

The following errors occur when there are Elasticsearch version compatibility issues.

Unsupported Elasticsearch version

Error message: "Support for Elasticsearch versions after their end-of-life (currently versions < 7.16) was removed. Using unsupported version of Elasticsearch may lead to unexpected and incorrect results."

Cause: The Elasticsearch version is no longer supported by the Grafana data source.

Solution:

1. Upgrade Elasticsearch to a supported version (7.17+, 8.x, or 9.x).
2. Refer to Elastic Product End of Life Dates for version support information.
3. Note that queries may still work, but Grafana does not guarantee functionality for unsupported versions.

Other common issues

The following issues don't produce specific error messages but are commonly encountered.

Empty query results

Cause: The query returns no data.

Solution:

1. Verify the time range includes data in your index.
2. Check the Lucene query syntax for errors.
3. Test the query directly in Elasticsearch using the _search API.
4. Ensure the index contains documents matching your query filters.

Slow query performance

Cause: Queries take a long time to execute.

Solution:

1. Reduce the time range of your query.
2. Add more specific filters to limit the data scanned.
3. Increase the date histogram interval.
4. Check Elasticsearch cluster performance and resource utilization.
5. Consider using index aliases or data streams for better query routing.

CORS errors in browser console

Cause: Cross-Origin Resource Sharing (CORS) is blocking requests from the browser to Elasticsearch.

Solution:

1. Use Server (proxy) access mode instead of Browser access mode in the data source configuration.
2. If Browser access is required, configure CORS settings in Elasticsearch:

http.cors.enabled: true
http.cors.allow-origin: '<your-grafana-url>'
http.cors.allow-headers: 'Authorization, Content-Type'
http.cors.allow-credentials: true

{{< admonition type="note" >}} Server (proxy) access mode is recommended for security and reliability. {{< /admonition >}}

Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. Check the Elasticsearch documentation for API-specific guidance.
2. Review the Grafana community forums for similar issues.
3. Contact Grafana Support if you have an Enterprise license.