Configure the Amazon CloudWatch data source

This document provides instructions for configuring the Amazon CloudWatch data source and explains available configuration options. For general information on adding and managing data sources, refer to Data source management.

Before you begin

• You must have the Organization administrator role to configure the CloudWatch data source. Organization administrators can also configure the data source via YAML with the Grafana provisioning system.

• Grafana comes with a built-in CloudWatch data source plugin, so you do not need to install a plugin.

• Familiarize yourself with your CloudWatch security configuration and gather any necessary security certificates, client certificates, and client keys.

Key concepts

If you're new to AWS, these terms are used throughout the configuration:

Add the CloudWatch data source

Complete the following steps to set up a new CloudWatch data source:

1. Click Connections in the left-side menu.
2. Click Add new connection
3. Type CloudWatch in the search bar.
4. Select the CloudWatch data source.
5. Click Add new data source in the upper right.

Grafana takes you to the Settings tab, where you will set up your CloudWatch configuration.

Configure the data source in the UI

The following are configuration options for the CloudWatch data source.

Grafana plugin requests to AWS are made on behalf of an AWS Identity and Access Management (IAM) role or IAM user. The IAM user or IAM role must have the associated policies to perform certain API actions.

For authentication options and configuration details, refer to AWS authentication.

Access & secret key:

You must use both an access key ID and a secret access key to authenticate.

Credentials file:

When you select the Credentials file authentication provider, you can specify which profile to read from the AWS shared credentials file.

Assume Role:

Additional Settings:

Proxy configuration:

The Proxy Configuration settings appear only when per-data source HTTP proxy support is enabled with the per_datasource_http_proxy_enabled option in the Grafana configuration file. They let each CloudWatch data source use its own outbound HTTP proxy for AWS requests.

CloudWatch Logs:

Application Signals trace link:

Grafana automatically creates a link to a trace in Application Signals data source if logs contain the @xrayTraceId field. To use this feature, you must already have an Application Signals data source configured. For details, refer to the Application Signals data source docs. To view the Application Signals link, select the log row in either the Explore view or dashboard Logs panel to view the log details section.

To log the @xrayTraceId, refer to the AWS Application Signals documentation. To provide the field to Grafana, your log queries must also contain the @xrayTraceId field, for example by using the query fields @message, @xrayTraceId.

Private data source connect - Only for Grafana Cloud users.

After configuring your Amazon CloudWatch data source options, click Save & test at the bottom to test the connection. When the test succeeds, Grafana confirms that it reached both CloudWatch APIs:

1. Successfully queried the CloudWatch metrics API.
2. Successfully queried the CloudWatch logs API.

{{< figure src="/media/docs/CloudWatch/CloudWatch-config-success-message.png" >}}

{{< admonition type="note" >}} To troubleshoot issues while setting up the CloudWatch data source, check the /var/log/grafana/grafana.log file. Common issues include invalid credentials, missing regions and metrics-only credentials. {{< /admonition >}}

IAM policy examples

To read CloudWatch metrics and EC2 tags, instances, regions, and alarms, you must grant Grafana permissions via IAM. You can attach these permissions to the IAM role or IAM user you configured in AWS authentication.

Metrics-only permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "CloudWatch:DescribeAlarmsForMetric",
        "CloudWatch:DescribeAlarmHistory",
        "CloudWatch:DescribeAlarms",
        "CloudWatch:ListMetrics",
        "CloudWatch:GetMetricData",
        "CloudWatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourceMetricsFromPerformanceInsights",
      "Effect": "Allow",
      "Action": "pi:GetResourceMetrics",
      "Resource": "*"
    }
  ]
}

Logs-only permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Metrics and logs permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "CloudWatch:DescribeAlarmsForMetric",
        "CloudWatch:DescribeAlarmHistory",
        "CloudWatch:DescribeAlarms",
        "CloudWatch:ListMetrics",
        "CloudWatch:GetMetricData",
        "CloudWatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourceMetricsFromPerformanceInsights",
      "Effect": "Allow",
      "Action": "pi:GetResourceMetrics",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Cross-account observability permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

{{< admonition type="note" >}} Cross-account observability lets you retrieve metrics and logs across different accounts in a single region, but you can't query EC2 Instance Attributes across accounts because those come from the EC2 API and not the CloudWatch API. {{< /admonition >}}

For more information on configuring authentication, refer to Configure AWS authentication.

Permissions reference

The following table explains why each permission is needed and whether it's required, so you can scope a policy to only the features you use. Grant cloudwatch:GetMetricData only for metrics, the logs: actions only for logs, and the EC2 and tag actions only if you use the related template variables. Granting a permission that you don't use has no effect on functionality.

When you use an Assume Role ARN, attach these query permissions to the assumed role. The primary credentials only need permission to perform sts:AssumeRole. For details, refer to Assume a role.

CloudWatch Logs data protection

CloudWatch Logs can protect data by applying log group data protection policies. When data protection is enabled for a log group, any sensitive data that matches the identifiers you select is automatically masked. To view masked data, your IAM role or user must have the logs:Unmask permission. For more details, refer to the AWS guide on masking sensitive log data.

Configure the data source with grafana.ini

The Grafana configuration file includes an AWS section where you can configure data source options:

Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning and available configuration options, refer to Provision Grafana.

Using AWS SDK (default):

apiVersion: 1
datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: default
      defaultRegion: eu-west-2

Using credentials' profile name (non-default):

apiVersion: 1

datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: credentials
      defaultRegion: eu-west-2
      customMetricsNamespaces: 'CWAgent,CustomNameSpace'
      profile: secondary

Using accessKey and secretKey:

apiVersion: 1

datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: keys
      defaultRegion: eu-west-2
    secureJsonData:
      accessKey: '<your access key>'
      secretKey: '<your secret key>'

Using AWS SDK Default and ARN of IAM Role to Assume:

apiVersion: 1
datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: default
      assumeRoleArn: arn:aws:iam::123456789012:root
      defaultRegion: eu-west-2

Configure with Terraform

You can configure the CloudWatch data source using Terraform with the Grafana Terraform provider.

For more information about provisioning resources with Terraform, refer to Grafana as code using Terraform.

The following example uses the Access and secret key authentication method:

resource "grafana_data_source" "cloudwatch" {
  name = "CloudWatch"
  type = "cloudwatch"

  json_data_encoded = jsonencode({
    authType      = "keys"
    defaultRegion = "eu-west-2"
  })

  secure_json_data_encoded = jsonencode({
    accessKey = "<YOUR_ACCESS_KEY>"
    secretKey = "<YOUR_SECRET_KEY>"
  })
}

The following example uses the AWS SDK Default authentication method with an assumed role:

resource "grafana_data_source" "cloudwatch" {
  name = "CloudWatch"
  type = "cloudwatch"

  json_data_encoded = jsonencode({
    authType      = "default"
    assumeRoleArn = "arn:aws:iam::123456789012:role/<ROLE_NAME>"
    defaultRegion = "eu-west-2"
  })
}

Replace the placeholders with your own values:

• <YOUR_ACCESS_KEY> and <YOUR_SECRET_KEY>: The AWS access key ID and secret access key for the Access and secret key method.
• <ROLE_NAME>: The name of the IAM role to assume.

For all available configuration options, refer to the Grafana provider data source resource documentation.