OpenTelemetry Architecture

Overview

Beads uses OpenTelemetry (OTel) for structured observability of all database operations, CLI commands, and Dolt version control. Telemetry is emitted via standard OTLP HTTP to any compatible backend (metrics, traces).

Backend-agnostic design: The system emits standard OpenTelemetry Protocol (OTLP) — any OTLP v1.x+ compatible backend can consume it. You are not obligated to use VictoriaMetrics/VictoriaLogs; these are simply development defaults.

Best-effort design: Telemetry initialization errors are returned but do not affect normal bd operation. The system remains functional even when telemetry is unavailable.

Implementation Status

Core Telemetry (Implemented ✅)

Dolt Backend Telemetry (Implemented ✅)

Server Lifecycle Telemetry (Not yet instrumented ❌)

internal/doltserver/ has no OTel imports. Server lifecycle spans and metrics are roadmap items (see Tier 1 below).

Roadmap

Current coverage: ~40% of the codebase. Below is a prioritized plan based on operational value vs. implementation effort.

Tier 1 — High value, moderate effort

Tracker integrations (internal/linear/, internal/jira/, internal/gitlab/)

External API calls are currently a black box. No visibility into latency, rate-limiting, or sync volume.

New metrics:

• bd_tracker_api_calls_total (Counter) — by tracker, method, status
• bd_tracker_api_latency_ms (Histogram) — by tracker, method
• bd_tracker_errors_total (Counter) — by tracker, error_type
• bd_tracker_issues_synced_total (Counter) — by tracker, direction

New spans: tracker.<name>.pull_issues, tracker.<name>.push_issue, tracker.<name>.resolve_state

Git operations (internal/git/)

Git push/pull can dominate wall-clock time but is currently invisible.

New metrics:

• bd_git_operation_duration_ms (Histogram) — by operation, status
• bd_git_errors_total (Counter) — by operation, error_type

New spans: git.clone, git.pull, git.push, git.commit, git.merge

Dolt server lifecycle (internal/doltserver/)

Server crashes and restarts are silent. No alerting possible.

New metrics:

• bd_doltserver_status (Gauge, 1=running/0=stopped)
• bd_doltserver_startup_ms (Histogram)
• bd_doltserver_restarts_total (Counter)
• bd_doltserver_errors_total (Counter) — by error_type

New spans: doltserver.start, doltserver.stop

Tier 2 — Medium value, low effort

Query engine (internal/query/)

Distinguishes whether slowness is client-side (parsing/compilation) or DB-side.

New spans: query.parse, query.compile New metrics: bd_query_duration_ms (Histogram), bd_query_parse_errors_total (Counter)

Validation engine (internal/validation/)

Data integrity errors are currently silent until they surface as user-visible failures.

New spans: validation.check_dependencies, validation.check_schema New metrics: bd_validation_errors_total (Counter) — by error_type

Dolt version control (internal/storage/dolt/versioned.go)

versioned.go has no OTel imports yet. Future spans:

• History: Query complete version history for an issue
• AsOf: Query state at specific commit or branch
• Diff: Cell-level diff between two commits
• ListBranches: Enumerate all branches
• GetCurrentCommit: Get HEAD commit hash
• GetConflicts: Check for merge conflicts

Dolt system table polling

Periodic SQL queries against Dolt system tables to surface metrics unavailable via OTLP (Dolt has no native OTel export):

Tier 3 — Low priority / future

• Command-level sub-spans: Instrument validation vs. DB vs. render breakdown per command (bd create, bd list, bd compact, etc.)
• Molecules & recipes: molecule.create, recipe.execute spans
• Hook duration metrics: Currently only spans (hook.exec), no histogram for aggregation
• OTel test suite: Integration tests that verify telemetry output (currently none)
• Lock wait recording: bd.db.lock_wait_ms histogram is registered but .Record() is not yet called

Components

1. Initialization (internal/telemetry/telemetry.go)

The telemetry.Init() function sets up OTel providers on process startup and returns only an error:

if err := telemetry.Init(ctx, "bd", version); err != nil {
    // Log and continue — telemetry is best-effort
}
defer telemetry.Shutdown(ctx)

Providers:

• Metrics: Any OTLP-compatible metrics backend via otlpmetrichttp exporter
• Traces: Stdout only (local debug). No remote trace backend in default stack.

Default endpoints (when BD_OTEL_METRICS_URL is not set):

• Metrics: http://localhost:8428/opentelemetry/api/v1/push
• Traces: stdout (via BD_OTEL_STDOUT=true)

Note: These defaults target VictoriaMetrics for local development convenience. Beads uses standard OTLP — you can override endpoints to use any OTLP v1.x+ compatible backend (Prometheus, Grafana Mimir, Datadog, New Relic, Grafana Cloud, Loki, OpenTelemetry Collector, etc.).

OTLP Compatibility:

• Uses standard OpenTelemetry Protocol (OTLP) over HTTP
• Protobuf encoding (VictoriaMetrics, Prometheus, and others accept this)
• Compatible with any backend that supports OTLP v1.x+

Resource attributes (set at init time):

• service.name: "bd"
• service.version: bd binary version
• host: system hostname
• os: system OS info

Custom resource attributes (via OTEL_RESOURCE_ATTRIBUTES env var or BEADS_ACTOR):

• bd.actor: Actor identity (from git config or env) — set after actor resolution
• bd.command: Current command name
• bd.args: Full arguments passed to command

2. Storage Instrumentation (internal/telemetry/storage.go)

The InstrumentedStorage wraps storage.Storage with OTel tracing and metrics:

• Every storage method gets a span
• Counters track operation counts
• Histograms track operation duration
• Error counters track failures

func WrapStorage(s storage.Storage) storage.Storage {
    if !Enabled() {
        return s  // Zero overhead when telemetry disabled
    }
    // Wrap with instrumentation
    return &InstrumentedStorage{inner: s, tracer, ops, dur, errs, issueGauge}
}

Metric names in code (OTel SDK notation with dots):

• bd.storage.operations → exported as bd_storage_operations_total by Prometheus/VM
• bd.storage.operation.duration → bd_storage_operation_duration_ms
• bd.storage.errors → bd_storage_errors_total
• bd.issue.count → bd_issue_count

Instrumented Storage Operations:

• Issue CRUD: CreateIssue, GetIssue, UpdateIssue, CloseIssue, DeleteIssue
• Dependencies: AddDependency, RemoveDependency, GetDependencies
• Labels: AddLabel, RemoveLabel, GetLabels
• Queries: SearchIssues, GetReadyWork, GetBlockedIssues
• Statistics: GetStatistics (also emits gauge of issue counts by status)
• Transactions: RunInTransaction

3. Dolt Backend Telemetry (internal/storage/dolt/store.go)

Dolt storage layer emits metrics for:

• bd.db.retry_count: SQL retries in server mode (recorded in withRetry when attempts > 1)
• bd.db.lock_wait_ms: Histogram registered but .Record() not yet called (stub)
• bd.db.circuit_trips: Circuit breaker trips to open state (recorded in withRetry)
• bd.db.circuit_rejected: Requests rejected by open circuit breaker (fail-fast path)
• SQL query spans via queryContext(), execContext(), queryRowContext() wrappers using doltTracer
• Dolt version control spans: dolt.commit, dolt.push, dolt.pull, dolt.merge, dolt.branch, dolt.checkout

SQL Span pattern (queryContext):

func (s *DoltStore) queryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error) {
    ctx, span := doltTracer.Start(ctx, "dolt.query",
        trace.WithSpanKind(trace.SpanKindClient),
        trace.WithAttributes(append(s.doltSpanAttrs(),
            attribute.String("db.operation", "query"),
            attribute.String("db.statement", spanSQL(query)),
        )...),
    )
    var rows *sql.Rows
    err := s.withRetry(ctx, func() error {
        rows, queryErr = s.db.QueryContext(ctx, query, args...)
        return queryErr
    })
    endSpan(span, wrapLockError(err))
    return rows, err
}

4. Dolt Version Control Telemetry (internal/storage/dolt/store.go)

Version control operations emit spans directly in store.go via doltTracer.Start(). These are not in versioned.go (which has no OTel imports).

Implemented spans (see Appendix for exact source locations):

• dolt.commit — CALL DOLT_COMMIT
• dolt.push — CALL DOLT_PUSH
• dolt.pull — CALL DOLT_PULL
• dolt.merge — CALL DOLT_MERGE
• dolt.branch — CALL DOLT_BRANCH
• dolt.checkout — CALL DOLT_CHECKOUT

5. Hook Telemetry (internal/hooks/)

Hooks emit a single root span per execution (hook.exec). There are no metric counters or histograms for hooks — only span-level observability. Duration metrics are a roadmap item (Tier 3).

Metric Naming Convention

OTel SDK uses dot-notation internally. Prometheus-compatible backends (VictoriaMetrics, Prometheus) export these as underscore-separated names with type suffixes:

Environment Variables

Beads-Level Variables

Context Variables

Dolt-Specific Variables (See DOLT.md)

Note: Dolt-specific configuration variables are documented in DOLT.md and are out of scope for OTEL design documentation.

Event Types

CLI Command Events

Storage Events

Dolt Events

Hooks Events

Monitoring Gaps

Currently Monitored ✅

Not Currently Monitored ❌

Queries

Metrics (Any OTLP-compatible backend)

Total counts by operation:

sum(rate(bd_storage_operations_total[5m])) by (db.operation)
sum(rate(bd_db_retry_count_total[5m]))

Latency distributions:

histogram_quantile(0.50, bd_storage_operation_duration_ms) by (db.operation)
histogram_quantile(0.95, bd_storage_operation_duration_ms) by (db.operation)
histogram_quantile(0.99, bd_storage_operation_duration_ms) by (db.operation)

Issue counts by status:

bd_issue_count{status="open"}
bd_issue_count{status="in_progress"}
bd_issue_count{status="closed"}
bd_issue_count{status="deferred"}

Dolt Telemetry Capabilities

Dolt Internal Metrics

Important: Dolt does not provide native OpenTelemetry export. The documentation search confirms there is no Dolt configuration variable or feature to enable OTLP export.

Dolt exposes internal metrics only via:

• performance_schema tables (MySQL standard, accessible via SQL queries)
• System tables (dolt_log, dolt_status, dolt_diff, dolt_branches, dolt_conflicts)

Beads implementation: Beads currently queries Dolt metrics via direct SQL (see cmd/bd/doctor/perf_dolt.go) rather than via OTLP. This is intentional — Dolt lacks native OTel support.

Dolt System Tables for Telemetry

Sample Queries for Dolt Telemetry

Commit frequency analysis:

SELECT
    DATE_FORMAT(commit_date, '%Y-%m') as month,
    COUNT(*) as commits
FROM dolt_log
GROUP BY month
ORDER BY month DESC;

Working set size tracking:

SELECT
    COUNT(*) as staged_changes,
    SUM(CASE WHEN staged = 1 THEN 1 ELSE 0 END) as added,
    SUM(CASE WHEN staged = 0 THEN 1 ELSE 0 END) as removed
FROM dolt_status;

Branch proliferation detection:

SELECT
    COUNT(*) as branch_count,
    MIN(commit_date) as oldest,
    MAX(commit_date) as newest
FROM dolt_branches;

Conflict analysis:

SELECT
    COUNT(*) as conflict_count,
    COUNT(DISTINCT table_name) as tables_affected
FROM dolt_conflicts;

Related Documentation

• OTel Data Model — Complete event schema
• OBSERVABILITY.md — Quick reference for metrics
• Dolt Backend — Dolt configuration and usage
• Dolt Concurrency — Concurrency model and transactions

Backends Compatible with OTLP

Production Recommendation: For production deployments, consider using OpenTelemetry Collector as a sidecar. The Collector provides:

• Single agent for all telemetry
• Advanced processing and batching
• Support for multiple backends simultaneously
• Better resource efficiency than per-process exporters

Appendix: Source Reference Audit

Audited against main @ 371df32b. All line numbers below refer to that commit.

Every factual claim in this document is backed by a specific source location. This table exists to prevent documentation drift and to make it easy to re-verify after code changes.

Initialization (internal/telemetry/telemetry.go, cmd/bd/main.go)

Storage Instrumentation (internal/telemetry/storage.go)

Dolt Backend (internal/storage/dolt/store.go)

versioned.go — no OTel

doltserver — no OTel

Hooks (internal/hooks/)

AI (internal/compact/haiku.go, cmd/bd/find_duplicates.go)