docs/sources/datasources/prometheus/configure/_index.md
This document provides instructions for configuring the Prometheus data source and explains the available configuration options. Grafana includes built-in support for Prometheus, so you don't need to install a plugin. For general information on adding a data source to Grafana, refer to Add a data source.
You need the Organization administrator role to configure the data source. You can also configure it via YAML using Grafana provisioning.
Grafana includes a built-in Prometheus data source; no plugin installation is required.
Know which type of Prometheus-compatible database you're connecting to (Prometheus, Mimir, Cortex, or Thanos), as the configuration options vary by type.
Have your Prometheus server URL ready. If using TLS/SSL, gather any necessary security certificates and client keys.
If using Basic authentication, have your username and password ready.
{{< shared id="add-prom-data-source" >}}
To add the Prometheus data source, complete the following steps:
Prometheus in the search bar.{{< /shared >}}
Grafana takes you to the Settings tab where you will set up your Prometheus configuration.
Following is a list of configuration options for Prometheus.
Connection:
http://localhost:9090. If it's hosted on a networked server, provide the server’s URL along with the port where Prometheus is running. Example: http://prometheus.example.orgname:9090.{{< admonition type="note" >}}
When running Grafana and Prometheus in separate containers, localhost refers to each container’s own network namespace. This means that localhost:9090 points to port 9090 inside the Grafana container, not on the host machine.
Use the IP address of the Prometheus container, or the hostname if you are using Docker Compose. Alternatively, you can use http://host.docker.internal:9090 to reference the host machine.
{{< /admonition >}}
{{< /shared >}}
Authentication:
There are three authentication options for the Prometheus data source.
Basic authentication - The most common authentication method.
Forward OAuth identity - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.
No authentication - Allows access to the data source without any authentication.
TLS settings:
{{< admonition type="note" >}} Use TLS (Transport Layer Security) for an additional layer of security when working with Prometheus. For information on setting up TLS encryption with Prometheus refer to Securing Prometheus API and UI Endpoints Using TLS Encryption. You must add TLS settings to your Prometheus configuration file prior to setting these options in Grafana. {{< /admonition >}}
HTTP headers:
Pass along additional information and metadata about the request or response.
Advanced settings:
Following are optional configuration settings you can configure for more control over your data source.
Alerting:
Manage alerts via Alerting UI -Toggled on by default. This enables data source-managed rules in Grafana Alerting for this data source. For Mimir, it enables managing data source-managed rules and alerts. For Prometheus, it only supports viewing existing rules and alerts, which are displayed as data source-managed. Change this by setting the default_manage_alerts_ui_toggle option in the grafana.ini configuration file.
Allow as recording rules target - Toggled on by default. This allows the data source to be selected as a target destination for writing Grafana-managed recording rules. When enabled, this data source will appear in the target data source list when creating or importing recording rules. When disabled, the data source will be filtered out from recording rules target selection. Change this by setting the default_allow_recording_rules_target_alerts_ui_toggle option in the grafana.ini configuration file.
Interval behavior:
15s. This interval determines how often Prometheus scrapes targets. Set it to match the typical scrape and evaluation interval in your Prometheus configuration file. If you set a higher value than your Prometheus configuration, Grafana will evaluate data at this interval, resulting in fewer data points.60s. Without a timeout, complex or inefficient queries can run indefinitely, consuming CPU and memory resources.Query editor:
Builder or Code. Builder mode helps you build queries using a visual interface. Code mode is geared for the experienced Prometheus user with prior expertise in PromQL. For more details on editor types refer to Prometheus query editor. You can switch easily code editors in the Query editor UI.Performance:
regex matching for label queries to improve performance, while others provide a metadata API. Setting this incorrectly may cause unexpected behavior when querying metrics and labels. Refer to your Prometheus documentation to ensure you select the correct type.Low, Medium, High, or None. Higher cache settings are recommended for high cardinality data sources.10m. This is a buffer of time added to incremental queries and this value is added to the duration of each incremental request.Other settings:
timeout, partial_response, dedup, or max_source_resolution. Multiple parameters should be joined using &.POST or GET HTTP method to query your data source. POSTis recommended and selected by default, as it supports larger queries. Select GET if you're using Prometheus version 2.1 or older, or if your network restricts POST requests.
Toggle onPOST method, whereas the label values endpoint only allows GET requests.Exemplars:
Support for exemplars is available only for the Prometheus data source. For more information on exemplars refer to Introduction to exemplars. An exemplar is a trace that represents a specific measurement taken within a given time interval.
Click the + sign to add exemplars.
Internal link) Defines the external link's URL trace backend. You can interpolate the value from the field by using the ${__value.raw} macro.Internal link is enabled.) Select the data source that the exemplar will link to from the drop-down.Label name field.labels object used to obtain the traceID property.You can add multiple exemplars.
Private data source connect - Only for Grafana Cloud users. Private data source connect, or PDC, allows you to establish a private, secured connection between a Grafana Cloud instance, or stack, and data sources secured within a private network. Click the drop-down to locate the URL for PDC. For more information regarding Grafana PDC refer to Private data source connect (PDC) and Configure Grafana private data source connect (PDC) for steps on setting up a PDC connection.
If you use PDC with SIGv4 (AWS Signature Version 4 Authentication), the PDC agent must allow internet egress tosts.<region>.amazonaws.com:443.
Click Manage private data source connect to open your PDC connection page and view your configuration details.
After you have configured your Prometheus data source options, click Save & test at the bottom to test out your data source connection.
You should see a confirmation dialog box that says:
Successfully queried the Prometheus API.
Next, you can start to visualize data by building a dashboard, or by querying data in the Explore view.
You can also remove a connection by clicking Delete.
You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to Provision Grafana.
{{< admonition type="note" >}} After you have provisioned a data source you cannot edit it. {{< /admonition >}}
Example of a Prometheus data source configuration:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://localhost:9090
jsonData:
httpMethod: POST
manageAlerts: true
allowAsRecordingRulesTarget: true
prometheusType: Prometheus
prometheusVersion: 3.3.0
cacheLevel: 'High'
disableRecordingRules: false
seriesEndpoint: false
timeInterval: 10s # Prometheus scrape interval
incrementalQueryOverlapWindow: 10m
exemplarTraceIdDestinations:
# Field with internal link pointing to data source in Grafana.
# datasourceUid value can be anything, but it should be unique across all defined data source uids.
- datasourceUid: my_jaeger_uid
name: traceID
# Field with external link.
- name: traceID
url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D'
The Prometheus data source works with Azure authentication. To configure Azure authentication refer to Configure Azure Active Directory (AD) authentication.
In Grafana Enterprise, you need to update the .ini configuration file. Refer to Configuration file location to locate your .ini file.
Add the following setting in the [auth] section of the .ini configuration file:
[auth]
azure_auth_enabled = true
{{< admonition type="note" >}}
If you are using Azure authentication, don't enable Forward OAuth identity. Both methods use the same HTTP authorization headers, and the OAuth token will override your Azure credentials.
{{< /admonition >}}
You can configure the Prometheus data source to disable recording rules in the data source configuration or provisioning file under disableRecordingRules in jsonData.
Refer to the following troubleshooting information as needed.
Data doesn't appear in Metrics Drilldown:
If you have successfully tested the connection to a Prometheus data source or are sending metrics to Grafana Cloud and there is no metric data appearing in Explore, make sure you've selected the correct data source from the data source drop-down menu. When using remote_write to send metrics to Grafana Cloud, the data source name follows the convention grafanacloud-stackname-prom.