Dashboard migrations

This document describes the Grafana dashboard migration system, focusing on conversion-level practices including metrics, logging, and testing infrastructure for API version conversions. For schema version migration implementation details, see the SchemaVersion Migration Guide.

Table of Contents

• Overview
  • Conversion Flow
  • v0 to v1 Conversion
  • v1 to v2 Conversion
  • v2 to v0/v1 Conversion
• Conversion Matrix
• API Versions
• Schema Versions
• Testing
  • Backend conversion tests
  • Backend and frontend conversion parity tests
• Monitoring Migrations
  • Metrics
    • Dashboard conversion success metric
    • Dashboard conversion failure metric
  • Error Types
  • Logging
    • Log structure
    • Log levels
  • Implementation Details
    • Automatic instrumentation
    • Error handling
  • Registration
    • Metrics registration
    • Available metrics
• Related Documentation

Overview

The Grafana dashboard migration system operates across three main conversion layers:

Conversion Flow: v0 → v1 → v2

v0alpha1 (Legacy JSON) → v1beta1 (Migrated JSON) → v2alpha1/v2beta1 (Structured)

v0 to v1 Conversion:

• All schema migrations (v0-v42) are executed in the backend
• Ports the logic from DashboardMigrator and implements built-in plugin migrations for panel plugins since backend cannot load plugins
• Transforms legacy JSON dashboards to migrated JSON format
• Handles backward compatibility for older dashboard formats
• See SchemaVersion Migration Guide for detailed instructions on creating new schema migrations

v1 to v2 Conversion:

• API version conversions between different Kubernetes API versions
• Transforms JSON dashboards to structured dashboard format
• v2 schema is the stable, typed schema with proper type definitions
• Handles modern dashboard features and Kubernetes-native storage
• Preserves Angular panel migration data for frontend processing (see Angular Panel Migrations)
• See V2 to V1 Layout Conversion for details on how V2 layouts are converted back to V1 panel arrays

v2 to v0/v1 Conversion:

• Converts structured v2 dashboards back to JSON format (v0alpha1 or v1beta1)
• Chains through intermediate versions: v2 → v1beta1 → v0alpha1
• v0alpha1 and v1beta1 share the same spec structure (only API version differs)
• Enables backward compatibility when storing v2 dashboards in legacy format

Angular Panel Migrations

When converting dashboards from v0/v1 to v2, panels with Angular types require special handling. The autoMigrateFrom field is used in v0 and v1 to indicate the panel was migrated from a deprecated plugin type, and the target plugin contains migration logic to transform the original panel's options and field configurations.

Panel plugins define their own migration logic via plugin.onPanelTypeChanged(). This migration runs in the frontend when a panel type changes, transforming old options/fieldConfig to the new format. Examples:

• singlestat.format: "short" → stat.fieldConfig.defaults.unit: "short"
• graph.legend.show: true → timeseries.options.legend.showLegend: true

The v2 schema doesn't include autoMigrateFrom as a typed field, so we need a mechanism to preserve the original panel data for the frontend to run these plugin migrations.

__angularMigration Temporary Data

The backend v1 → v2 conversion preserves the original panel data in a temporary field within vizConfig.spec.options. This works for any Angular panel, not just specific panel types:

{
  "vizConfig": {
    "kind": "stat",
    "spec": {
      "options": {
        "__angularMigration": {
          "autoMigrateFrom": "singlestat",
          "originalPanel": {
            "type": "singlestat",
            "format": "short",
            "colorBackground": true,
            "sparkline": { "show": true },
            "fieldConfig": { ... },
            "options": { ... }
          }
        }
      }
    }
  }
}

How It Works

1. Backend (v1 → v2): The conversion detects panels that need Angular migration in two ways:

   • If autoMigrateFrom is already set on the panel (from v0 → v1 migration) - panel type already converted
   • If the panel type is a known Angular panel type (fallback for dashboards stored directly as v1 without v0 → v1 migration)

   In the second case, the conversion also transforms the panel type (e.g., singlestat → stat) and sets autoMigrateFrom. This replicates the same logic as the v0 → v1 migration.

   The entire original panel is stored under options.__angularMigration for the frontend to run plugin-specific migrations.

2. Frontend (v2 load): When building a VizPanel from v2 data:

   • Extracts __angularMigration from options
   • Removes it from the options object (not persisted)
   • Attaches a custom migration handler via _UNSAFE_customMigrationHandler

3. Plugin load (VizPanel activation): When the plugin loads, the migration handler calls plugin.onPanelTypeChanged() with the original panel data, allowing the plugin's own migration code to run

Key Points

• Works for any panel with autoMigrateFrom, not limited to specific panel types
• Each plugin defines its own migration logic - the backend just preserves the data
• The originalPanel contains the complete panel data to ensure no information is lost
• Migration data is removed from options after loaded in frontend
• The 0v → v1 and v1 → v2 conversions automatically detects these Angular panel types if autoMigrateFrom is not set.

Implementation Files

Conversion Matrix

The system supports conversions between all dashboard API versions:

Each conversion path is automatically instrumented with metrics and logging.

API Versions

The supported dashboard API versions are:

• dashboard.grafana.app/v0alpha1 - Legacy JSON dashboard format
• dashboard.grafana.app/v1beta1 - Migrated JSON dashboard format
• dashboard.grafana.app/v2alpha1 - New structured dashboard format
• dashboard.grafana.app/v2beta1 - Enhanced structured dashboard format

Schema Versions

Schema versions (v13-v42) apply only to v0alpha1 and v1beta1 dashboards:

• Minimum supported version: v13
• Latest version: v42
• Migration path: Sequential (v13→v14→v15...→v42)

For detailed information about creating schema version migrations, see the SchemaVersion Migration Guide.

Testing

The implementation includes comprehensive test coverage for conversion-level operations:

• Backend conversion tests: API version conversions with metrics validation
• Backend migration tests: Schema version migration tests
• Frontend parity tests: TypeScript conversion tests that compare against backend golden files
• Metrics tests: Prometheus counter validation

Golden files

Both the migration and conversion test suites produce golden files -- JSON snapshots of their output. These files are not tracked in git; only their SHA-256 checksums are committed (in testdata/golden_checksums.json within each package). Tracking checksums instead of full files keeps the repository small (the golden files total hundreds of megabytes) and avoids noisy diffs when migration logic changes, while still catching unintended output changes. The golden files are generated on demand by running the Go tests, and in CI they are built as artifacts so the frontend parity tests can consume them.

The shared checksum logic lives in testutil/golden.go (ChecksumStore). Each package wires it via TestMain in its own golden_test.go.

Generating golden files locally:

# From apps/dashboard/
make generate-golden-files

This runs the migration tests first (which produce testdata/output/ and testdata/dev-dashboards-output/), then the conversion tests (which produce conversion/testdata/output/ and conversion/testdata/migrated_dashboards_output/).

Updating checksums after changing migrations or conversions:

# From apps/dashboard/
make regenerate-golden-checksums

This sets REGENERATE_CHECKSUMS=true and re-runs both test suites so the golden_checksums.json files are updated with the new hashes. Commit the updated checksum files.

How validation works at test time:

1. The test always writes the golden file to disk (so frontend tests can read it).
2. It computes the SHA-256 of the written data.
3. In normal mode, it compares against the stored checksum and fails on mismatch.
4. In regenerate mode (REGENERATE_CHECKSUMS=true), it records the new hash instead.

Backend conversion tests

The backend conversion tests validate API version conversions and metrics instrumentation:

• API conversion tests: Test conversions between v0alpha1, v1beta1, v2alpha1, v2beta1
• Metrics validation: Tests verify that conversion metrics are properly recorded
• Error handling: Tests validate error classification and logging

Test execution:

# All backend conversion tests
go test ./apps/dashboard/pkg/migration/conversion/... -v

# Metrics validation tests
go test ./apps/dashboard/pkg/migration/... -run TestSchemaMigrationMetrics

Backend and frontend conversion parity tests

These tests ensure that backend (Go) and frontend (TypeScript) conversions produce identical outputs. This is critical because:

• Dual implementation: Both backend and frontend implement dashboard version conversions
• Consistency requirement: Users should see the same dashboard regardless of which path is used
• API flexibility: The API may return dashboards in different versions depending on context

The frontend tests read golden files generated by the backend tests. In CI, a dedicated generate-golden-files job runs the Go tests and uploads the output as a build artifact, which the frontend test jobs download before running.

Why normalize through Scene?

Both backend and frontend outputs are passed through the same Scene load/save cycle before comparison. This normalization:

• Eliminates differences from default values added by Scene
• Handles field ordering variations
• Simulates the real-world flow: dashboard loaded -> edited -> saved

Test locations:

Test execution:

# V1 to V2 conversion parity tests
yarn test transformSaveModelV1ToV2.test.ts

# V2 to V1 conversion parity tests
yarn test transformSaveModelV2ToV1.test.ts

# Schema migration parity tests
yarn test DashboardMigratorToBackend.test.ts

Test approach (v1 -> v2):

• Backend path: v1beta1 -> Go conversion -> v2beta1 -> Scene -> normalized output
• Frontend path: v1beta1 -> Scene -> v2beta1 -> Scene -> normalized output
• Test data: Uses golden files from apps/dashboard/pkg/migration/conversion/testdata/ and migrated dashboards

Test approach (v2 -> v1/v0):

• Backend path: v2beta1 -> Go conversion -> v1beta1/v0alpha1 -> Scene -> normalized output
• Frontend path: v2beta1 -> Scene -> v1beta1 -> Scene -> normalized output
• Target versions: Tests both v0alpha1 and v1beta1 (they share the same spec structure)
• Test data: Uses golden files from apps/dashboard/pkg/migration/conversion/testdata/

For schema version migration testing details, see the SchemaVersion Migration Guide.

Monitoring Migrations

The dashboard migration system provides comprehensive observability through metrics, logging, and error classification to monitor conversion operations.

Metrics

The dashboard migration system now provides comprehensive observability through:

• Prometheus metrics for tracking conversion success/failure rates and performance
• Structured logging for debugging and monitoring conversion operations
• Automatic instrumentation via wrapper functions that eliminate code duplication
• Error classification to distinguish between different types of migration failures

1. Dashboard conversion success metric

Metric Name: grafana_dashboard_migration_conversion_success_total

Type: Counter

Description: Total number of successful dashboard conversions

Labels:

• source_version_api - Source API version (e.g., "dashboard.grafana.app/v0alpha1")
• target_version_api - Target API version (e.g., "dashboard.grafana.app/v1beta1")
• source_schema_version - Source schema version (e.g., "16") - only for v0/v1 dashboards
• target_schema_version - Target schema version (e.g., "41") - only for v0/v1 dashboards

Example:

grafana_dashboard_migration_conversion_success_total{
  source_version_api="dashboard.grafana.app/v0alpha1",
  target_version_api="dashboard.grafana.app/v1beta1",
  source_schema_version="16",
  target_schema_version="41"
} 1250

2. Dashboard conversion failure metric

Metric Name: grafana_dashboard_migration_conversion_failure_total

Type: Counter

Description: Total number of failed dashboard conversions

Labels:

• source_version_api - Source API version
• target_version_api - Target API version
• source_schema_version - Source schema version (only for v0/v1 dashboards)
• target_schema_version - Target schema version (only for v0/v1 dashboards)
• error_type - Classification of the error (see Error Types section)

Example:

grafana_dashboard_migration_conversion_failure_total{
  source_version_api="dashboard.grafana.app/v0alpha1",
  target_version_api="dashboard.grafana.app/v1beta1",
  source_schema_version="14",
  target_schema_version="41",
  error_type="schema_version_migration_error"
} 42

Error Types

The error_type label classifies failures into four categories:

1. conversion_error

• General conversion failures not related to schema migration
• API-level conversion issues
• Programming errors in conversion functions

2. schema_version_migration_error

• Failures during individual schema version migrations (v14→v15, v15→v16, etc.)
• Schema-specific transformation errors
• Data format incompatibilities

3. schema_minimum_version_error

• Dashboards with schema versions below the minimum supported version (< v13)
• These are logged as warnings rather than errors
• Indicates dashboards that cannot be migrated automatically

4. conversion_data_loss_error

• Data loss detected during conversion
• Automatically checks that panels, queries, annotations, and links are preserved
• Triggered when target has fewer items than source
• Includes detailed loss metrics in logs (see Data Loss Detection)

Logging

Log structure

All migration logs use structured logging with consistent field names:

Base Fields (always present):

• sourceVersionAPI - Source API version
• targetVersionAPI - Target API version
• dashboardUID - Unique identifier of the dashboard being converted

Schema Version Fields (v0/v1 dashboards only):

• sourceSchemaVersion - Source schema version number
• targetSchemaVersion - Target schema version number
• erroredSchemaVersionFunc - Name of the schema migration function that failed (on error)

Error Fields (failures only):

• errorType - Same classification as metrics error_type label
• erroredConversionFunc - Name of the conversion function that failed
• error - The actual error message

Data Loss Fields (conversion_data_loss_error only):

• panelsLost - Number of panels lost
• queriesLost - Number of queries lost
• annotationsLost - Number of annotations lost
• linksLost - Number of links lost
• variablesLost - Number of template variables lost

Log levels

Success (DEBUG level)

{
  "level": "debug",
  "msg": "Dashboard conversion succeeded",
  "sourceVersionAPI": "dashboard.grafana.app/v0alpha1",
  "targetVersionAPI": "dashboard.grafana.app/v1beta1",
  "dashboardUID": "abc123",
  "sourceSchemaVersion": 16,
  "targetSchemaVersion": 41
}

Conversion/Migration Error (ERROR level)

{
  "level": "error", 
  "msg": "Dashboard conversion failed",
  "sourceVersionAPI": "dashboard.grafana.app/v0alpha1",
  "targetVersionAPI": "dashboard.grafana.app/v1beta1",
  "erroredConversionFunc": "Convert_V0_to_V1",
  "dashboardUID": "abc123",
  "sourceSchemaVersion": 16,
  "targetSchemaVersion": 41,
  "erroredSchemaVersionFunc": "V24",
  "errorType": "schema_version_migration_error",
  "error": "migration failed: table panel plugin not found"
}

Minimum Version Error (WARN level)

{
  "level": "warn",
  "msg": "Dashboard conversion failed", 
  "sourceVersionAPI": "dashboard.grafana.app/v0alpha1",
  "targetVersionAPI": "dashboard.grafana.app/v1beta1",
  "erroredConversionFunc": "Convert_V0_to_V1",
  "dashboardUID": "def456",
  "sourceSchemaVersion": 10,
  "targetSchemaVersion": 41,
  "erroredSchemaVersionFunc": "",
  "errorType": "schema_minimum_version_error",
  "error": "dashboard schema version 10 cannot be migrated"
}

Data Loss Error (ERROR level)

{
  "level": "error",
  "msg": "Dashboard conversion failed",
  "sourceVersionAPI": "dashboard.grafana.app/v1beta1",
  "targetVersionAPI": "dashboard.grafana.app/v2alpha1",
  "erroredConversionFunc": "V1beta1_to_V2alpha1",
  "dashboardUID": "abc123",
  "sourceSchemaVersion": 42,
  "targetSchemaVersion": 42,
  "panelsLost": 0,
  "queriesLost": 2,
  "annotationsLost": 0,
  "linksLost": 0,
  "variablesLost": 0,
  "errorType": "conversion_data_loss_error",
  "error": "data loss detected: query count decreased from 7 to 5"
}

Data Loss Detection

Automatic Runtime Checks:

Every conversion automatically detects data loss by comparing:

• Panel count - Visualization panels (regular + library panels)
• Query count - Data source queries (excludes invalid row panel queries)
• Annotation count - Dashboard-level annotations
• Link count - Navigation links
• Variable count - Template variables (from templating.list in v0/v1, variables in v2)

Detection Logic:

• ✅ Allows additions: Default annotations, enriched data
• ❌ Detects losses: Any decrease in counts triggers conversion_data_loss_error

Testing:

Run comprehensive data loss tests on all conversion test files:

# Test all conversions for data loss
go test ./apps/dashboard/pkg/migration/conversion/... -run TestDataLossDetectionOnAllInputFiles -v

# Test shows detailed panel/query analysis when loss is detected

Implementation: See conversion/conversion_data_loss_detection.go and conversion/README.md for details.

Implementation Details

Automatic instrumentation

All dashboard conversions are automatically instrumented via the normalizeConversion wrapper function, which provides:

• Metrics tracking: Records success/failure metrics with proper labels
• Status management: Ensures storedVersion is preserved through multi-step conversions
• Error handling: Sets conversion status on both success and failure
• Metadata copying: Copies ObjectMeta, APIVersion, and Kind on error
• Default spec ensuring: Ensures valid default specs/layouts on error to prevent JSON marshaling issues
• Data loss detection: Compares panel/query counts before and after conversion

The normalizeConversion wrapper uses the DashboardConversion interface, which all dashboard API versions implement to provide consistent conversion behavior:

// All conversion functions are wrapped with normalizeConversion
// This ensures consistent status handling, metrics, and error management
s.AddConversionFunc((*dashv0.Dashboard)(nil), (*dashv1.Dashboard)(nil),
    normalizeConversion(dashv0.APIVERSION, dashv1.APIVERSION, func(a, b interface{}, scope conversion.Scope) error {
        return Convert_V0_to_V1beta1(a.(*dashv0.Dashboard), b.(*dashv1.Dashboard), scope)
    }))

Error handling

Custom error types provide structured error information:

// Schema migration errors
type MigrationError struct {
    msg            string
    targetVersion  int
    currentVersion int
    functionName   string
}

// API conversion errors  
type ConversionError struct {
    msg               string
    functionName      string
    currentAPIVersion string
    targetAPIVersion  string
}

// Data loss errors are detected when dashboard components (panels, queries, annotations, links, variables) 
// are lost during conversion
type ConversionDataLossError struct {
    functionName     string  // Function where data loss was detected (e.g., "V1_to_V2alpha1")
    message          string  // Detailed error message with loss statistics
    sourceAPIVersion string  // Source API version (e.g., "dashboard.grafana.app/v1beta1")
    targetAPIVersion string  // Target API version (e.g., "dashboard.grafana.app/v2alpha1")
}

Registration

Metrics registration

Metrics must be registered with Prometheus during service initialization:

import "github.com/grafana/grafana/apps/dashboard/pkg/migration"

// Register metrics with Prometheus
migration.RegisterMetrics(prometheusRegistry)

Available metrics

The following metrics are available after registration:

// Success counter
migration.MDashboardConversionSuccessTotal

// Failure counter  
migration.MDashboardConversionFailureTotal

Related Documentation

• Schema Migration Guide - Complete guide for creating new dashboard schema migrations
• V2 to V1 Layout Conversion - How V2 layouts (GridLayout, RowsLayout, TabsLayout, AutoGridLayout) are converted to V1 panel arrays
• PR #110178 - Dashboard migration: Add missing metrics registration