OpenTSDB annotations

Annotations allow you to overlay event information on graphs, providing context for metric changes. The OpenTSDB data source supports both metric-specific annotations and global annotations stored in OpenTSDB.

For general information about annotations in Grafana, refer to Annotate visualizations.

Annotation types

OpenTSDB supports two types of annotations:

How Grafana retrieves annotations

When you configure an annotation query, Grafana queries OpenTSDB for the specified metric and retrieves any annotations associated with that metric's time series. The query includes the globalAnnotations=true parameter, which allows Grafana to also retrieve global annotations when enabled.

Grafana displays the description field from each annotation as the annotation text.

Configure an annotation query

To add OpenTSDB annotations to a dashboard:

1. Click the dashboard settings icon (gear) in the top navigation.
2. Select Annotations in the left menu.
3. Click Add annotation query.
4. Select the OpenTSDB data source.
5. Configure the annotation query fields as described in the following table.
6. Click Save dashboard.

Annotation query fields

Example annotation queries

The following examples demonstrate common annotation use cases.

Track application deployments

Monitor when deployments occur for a specific application:

This query retrieves annotations attached to the deploy.myapp metric, showing deployment events for that specific application.

Monitor infrastructure-wide events

Capture system-wide events such as network changes or datacenter maintenance:

This query retrieves both metric-specific and global annotations, providing a complete picture of infrastructure events.

Track incidents and outages

Mark incident start and resolution times:

Monitor configuration changes

Track when configuration changes are applied:

Correlate multiple event types

You can add multiple annotation queries to a single dashboard to correlate different event types. For example:

1. Add a "Deployments" annotation query for deploy.* metrics.
2. Add an "Incidents" annotation query for events.incident.
3. Add a "Maintenance" annotation query with global annotations enabled.

This allows you to see how deployments, incidents, and maintenance windows relate to your metric data.

How annotations appear

Annotations appear as vertical lines on time series panels at the timestamps where events occurred. Hover over an annotation marker to view:

• The annotation name (from your query configuration)
• The event description (from the OpenTSDB annotation's description field)
• The timestamp

Different annotation queries can be assigned different colors in the dashboard settings to distinguish between event types.

Create annotations in OpenTSDB

To display annotations in Grafana, you must first create them in OpenTSDB. OpenTSDB provides an HTTP API for managing annotations.

Annotation data structure

OpenTSDB annotations have the following fields:

Create a global annotation

Use the OpenTSDB API to create a global annotation:

curl -X POST http://<OPENTSDB_HOST>:4242/api/annotation \
  -H "Content-Type: application/json" \
  -d '{
    "startTime": 1609459200,
    "description": "Production deployment v2.5.0",
    "notes": "Deployed new feature flags and performance improvements",
    "custom": {
      "version": "2.5.0",
      "environment": "production",
      "deployer": "jenkins"
    }
  }'

Create a metric-specific annotation

To attach an annotation to a specific time series, include the tsuid:

curl -X POST http://<OPENTSDB_HOST>:4242/api/annotation \
  -H "Content-Type: application/json" \
  -d '{
    "startTime": 1609459200,
    "endTime": 1609462800,
    "tsuid": "000001000001000001",
    "description": "Server maintenance",
    "notes": "Scheduled maintenance window for hardware upgrade"
  }'

To find the TSUID for a metric, use the OpenTSDB /api/uid/tsmeta endpoint.

Create annotations programmatically

Integrate annotation creation into your deployment pipelines or monitoring systems:

Deployment script example:

#!/bin/bash
VERSION=$1
TIMESTAMP=$(date +%s)

curl -X POST http://opentsdb.example.com:4242/api/annotation \
  -H "Content-Type: application/json" \
  -d "{
    \"startTime\": $TIMESTAMP,
    \"description\": \"Deployed version $VERSION\",
    \"custom\": {
      \"version\": \"$VERSION\",
      \"environment\": \"production\"
    }
  }"

For more details on the annotation API, refer to the OpenTSDB annotation API documentation.

Troubleshoot annotation issues

The following section addresses common issues you may encounter when using OpenTSDB annotations.

Annotations don't appear

Possible causes and solutions:

Annotation text is empty

The annotation displays but has no description text.

Solution: Ensure the description field is populated when creating annotations in OpenTSDB. Grafana displays the description field as the annotation text.

Next steps

• Build queries to visualize metrics alongside annotations.
• Use template variables to create dynamic dashboards.
• Set up alerting to get notified when metrics cross thresholds.