OpenTelemetry Architecture

Overview

Gas Town uses OpenTelemetry (OTel) for structured observability of all agent operations. Telemetry is emitted via standard OTLP HTTP to any compatible backend (metrics, logs).

Backend-agnostic design: The system emits standard OpenTelemetry Protocol (OTLP) — any OTLP v1.x+ compatible backend can consume it. You are not obligated to use VictoriaMetrics/VictoriaLogs; these are simply development defaults.

Best-effort design: Telemetry initialization errors are returned but do not affect normal GT operation. The system remains functional even when telemetry is unavailable.

Quick Setup

Set at least one endpoint variable to activate telemetry — both endpoints unset means telemetry is completely disabled (no instrumentation code runs):

# Full local setup (recommended)
export GT_OTEL_METRICS_URL=http://localhost:8428/opentelemetry/api/v1/push
export GT_OTEL_LOGS_URL=http://localhost:9428/insert/opentelemetry/v1/logs

# Opt-in features
export GT_LOG_BD_OUTPUT=true      # Include bd stdout/stderr in bd.call records
export GT_LOG_AGENT_OUTPUT=true   # Stream Claude conversation turns to logs (PR #2199)

Local backends (Docker):

docker run -d -p 8428:8428 victoriametrics/victoria-metrics
docker run -d -p 9428:9428 victoriametrics/victoria-logs

Verify: gt prime should emit a prime event visible at http://localhost:9428/select/vmui.

Implementation Status

Core Telemetry (Main ✅)

Session Lifecycle (Main ✅)

Workflow & Work Events (Main ✅)

Agent Lifecycle (Main ✅)

Molecule Lifecycle

Agent Events (PR #2199)

Activation (PR #2199): Requires GT_LOG_AGENT_OUTPUT=true AND GT_OTEL_LOGS_URL set.

Roadmap

P0 — Critical (blocking accurate attribution)

Work context injection at gt prime — implemented in PR #2199

Polecats are generic agents — they have no fixed rig. GT_RIG at session start reflects an allocation rig or is empty, which is meaningless for attributing work. The actual work context (which rig, bead, and molecule a polecat is processing) is only known at each gt prime invocation.

A single polecat session goes through multiple gt prime cycles, each on a potentially different rig and bead:

session start → rig="" (generic, no work yet)
gt prime #1   → work_rig="gastown", work_bead="sg-05iq", work_mol="mol-polecat-work"
  bd.call, mail, sling, done  ← carry work context from prime #1
gt prime #2   → work_rig="sfgastown", work_bead="sg-g8vs", work_mol="mol-polecat-work"
  bd.call, mail, sling, done  ← carry work context from prime #2

Fix: at each gt prime, inject GT_WORK_RIG, GT_WORK_BEAD, GT_WORK_MOL into the tmux session environment (via SetEnvironment), not just emit them as log attributes. This ensures all subsequent subprocesses (bd, mail, agent logging) inherit the current work context automatically until the next prime overwrites it.

New attributes emitted on the prime event and carried by all events until the next prime:

P1 — High value

Token cost metric (gastown.token.cost_usd)

Compute dollar cost per run from token counts using Claude model pricing. Emit as a Gauge metric at session end. Enables per-rig and per-bead cost dashboards.

New metric: gastown.token.cost_usd{rig, role, agent_type} — accumulated cost per session. New event attribute on agent.usage: cost_usd — cost of the current turn.

Go runtime/process metrics (low effort)

The OTel Go SDK has a contrib package (go.opentelemetry.io/contrib/instrumentation/runtime) that auto-emits goroutine count, GC pause duration, heap usage, and memory allocations. Activation is ~5 lines of code in telemetry.Init().

New metrics: process.runtime.go.goroutines, process.runtime.go.gc.pause_ns, process.runtime.go.mem.heap_alloc_bytes

Refinery queue telemetry

The Refinery's merge queue is a central health indicator but currently completely dark to observability. Expose:

New log event: refinery.dispatch with bead_id, queue_depth, wait_ms, status.

Distributed Traces (OTel Traces SDK)

Currently the waterfall relies on run.id as a manual correlation key across flat log records. Replacing this with proper OTel Traces would enable:

• Visual waterfall in Jaeger / Grafana Tempo
• Automatic parent → child span attribution (no manual run.id joins)
• P95/P99 latency per operation derived from spans, not histograms

Architecture: each polecat session spawn creates a root span (gt.session). Child spans are created for bd.call, mail, sling, done. The run.id becomes the trace ID. GT_RUN propagation becomes W3C traceparent header injection.

This is a significant effort (requires go.opentelemetry.io/otel/trace + tracer provider + exporter) but would be the single highest-impact observability improvement.

P2 — Medium value

Scheduler dispatch telemetry

Expose the capacity-controlled dispatch cycle:

flowchart TD
    A[Query Pending
Readiness Filter] --> B{Capacity Available?}
    B -->|Yes| C[Plan Dispatch
ToDispatch array]
    B -->|No| D[Skip All
Reason: capacity]
    C --> E[Execute Dispatch
gt sling / gt prime]
    E --> F{Dispatch Result}
    F -->|Success| G[Close Sling Context
OnSuccess]
    F -->|Failure| H[Retry / Quarantine
OnFailure]
    G --> I[Log: scheduler.dispatch_cycle
dispatched++, status=ok]
    H --> I
    D --> I
    I --> J[Log: scheduler.queue_depth
current pending count]
    J --> K[Log: scheduler.capacity_usage
free/used slots]

New metrics: scheduler.dispatch_cycle (dispatched/failed/skipped counts), scheduler.queue_depth (histogram), scheduler.capacity_usage (gauge).

done event enrichment

Currently done carries only exit_type and status. Adding work context enables per-rig completion analysis:

New attributes: rig, bead_id, time_to_complete_ms (wall time from session start to done).

Witness patrol cycle telemetry

Each witness patrol cycle should emit: duration, stale sessions detected, restarts triggered. Enables trend analysis on witness health.

New event: witness.patrol with duration_ms, stale_count, restart_count, status.

Dolt health metrics

Dolt issues are only detected at spawn time today. Exposing health metrics continuously:

New metrics: gastown.dolt.connections, gastown.dolt.query_duration_ms (histogram), gastown.dolt.replication_lag_ms.

P3 — Nice to have

Components

1. Initialization (internal/telemetry/telemetry.go)

The telemetry.Init() function sets up OTel providers on process startup:

provider, err := telemetry.Init(ctx, "gastown", version)
if err != nil {
    // Log and continue — telemetry is best-effort
}
defer provider.Shutdown(ctx)

Exact signature: func Init(ctx context.Context, serviceName, serviceVersion string) (*Provider, error)

Providers:

• Metrics: Any OTLP-compatible metrics backend via otlpmetrichttp exporter
• Logs: Any OTLP-compatible logs backend via otlploghttp exporter

Default endpoints (when GT_OTEL_* variables are not set):

• Metrics: http://localhost:8428/opentelemetry/api/v1/push
• Logs: http://localhost:9428/insert/opentelemetry/v1/logs

Note: These defaults target VictoriaMetrics/VictoriaLogs for local development convenience. Gas Town uses standard OTLP — you can override endpoints to use any OTLP v1.x+ compatible backend (Prometheus, Grafana Mimir, Datadog, New Relic, Grafana Cloud, Loki, OpenTelemetry Collector, etc.).

OTLP Compatibility:

• Uses standard OpenTelemetry Protocol (OTLP) over HTTP
• Protobuf encoding (VictoriaMetrics, Prometheus, and others accept this)
• Compatible with any backend that supports OTLP v1.x+

Resource attributes (set at init time by the OTel SDK):

Custom resource attributes (via OTEL_RESOURCE_ATTRIBUTES env var, set by SetProcessOTELAttrs()):

2. Recording Layer (internal/telemetry/recorder.go)

The recorder provides type-safe functions for emitting all GT telemetry events. Each function emits:

1. OTel metric counter (→ VictoriaMetrics, aggregated)
2. OTel log record (→ VictoriaLogs, full detail)

run.id on log records: On main, log records do not carry run.id. After PR #2199 merges, addRunID(ctx, &r) will be called on every log record, injecting run.id from context (set via WithRunID) or from the GT_RUN env var.

Recording Pattern

func RecordSomething(ctx context.Context, args ..., err error) {
    initInstruments() // Lazy-load OTel instruments
    status := statusStr(err) // "ok" or "error"
    inst.somethingTotal.Add(ctx, 1, metric.WithAttributes(
        attribute.String("status", status),
        attribute.String("label", value),
    ))
    emit(ctx, "something", severity(err),
        otellog.String("key1", value1),
        otellog.String("key2", value2),
        otellog.String("status", status),
        errKV(err), // Empty string or error message
    )
}

Instrument Types

3. Context Propagation

Subprocess Integration (internal/telemetry/subprocess.go)

Two mechanisms ensure subprocess telemetry is correlated:

1. Process-level inheritance (SetProcessOTELAttrs):

• Called once at GT startup
• Sets OTEL_RESOURCE_ATTRIBUTES in process environment
• All exec.Command() subprocesses inherit these env vars automatically

2. Manual injection (OTELEnvForSubprocess):

• For callers building cmd.Env explicitly (overriding os.Environ)
• Returns pre-built env slice with:
  • OTEL_RESOURCE_ATTRIBUTES (GT context attributes)
  • BD_OTEL_METRICS_URL (mirrors GT_OTEL_METRICS_URL)
  • BD_OTEL_LOGS_URL (mirrors GT_OTEL_LOGS_URL)
  • GT_RUN (run ID for correlation — PR #2199)

Run ID Correlation (PR #2199)

On main, there is no run-level correlation key in log records. PR #2199 adds:

• GT_RUN env var — UUID generated at polecat spawn
• gt.run_id in OTEL_RESOURCE_ATTRIBUTES — carried by all subprocesses
• WithRunID(ctx, runID) / RunIDFromCtx(ctx) — Go context carrier
• addRunID(ctx, &record) — called in every emit, injects run.id into log record

Query example (after PR #2199): Retrieve all events for a single session run

run.id:uuid-1234

4. Agent Logging (PR #2199)

Status: PR #2199 (otel-p0-work-context) — not on main. The files below are added in PR #2199 and do not exist at origin/main.

Opt-in feature: GT_LOG_AGENT_OUTPUT=true streams native AI agent JSONL to VictoriaLogs.

How it works:

1. ActivateAgentLogging() (internal/session/agent_logging_unix.go) spawns detached gt agent-log process
2. Uses Setsid so it survives parent process exit
3. PID file at /tmp/gt-agentlog-<session>.pid ensures single instance
4. --since=now-60s filters to only this session's Claude instance
5. gt agent-log (internal/cmd/agent_log.go) tails JSONL files and emits RecordAgentEvent for each
6. internal/agentlog/ package — adapters for claudecode and opencode JSONL formats

Events emitted:

• agent.event: One record per conversation turn (text, tool_use, tool_result, thinking)
• agent.usage: Token usage per assistant turn (input, output, cache stats)

Session name in telemetry:

• session: Tmux session name (e.g., gt-gastown-Toast)
• native_session_id: Claude Code JSONL filename UUID

Environment Variables

GT-Level Variables

Session Context Variables (Set by session.StartSession)

Event Types

See OTel Data Model for the complete event schema, attribute tables, and metric reference.

Monitoring Gaps

Currently Monitored ✅

Not Currently Monitored ❌

Generic Polecat Work Context ⚠️

Critical gap: Polecats are generic agents with no fixed rig. gt.rig in resource attributes reflects the allocation rig (or is empty), which has no bearing on the actual work being done. Work context is only determined at each gt prime invocation — and changes with every new work assignment.

This means all events emitted between two gt prime calls (bd.call, mail, sling, done) have no work attribution today. You cannot answer "which bead did this bd.call serve?" from current telemetry.

Impact:

• gt.rig resource attribute is the allocation rig, not the work rig — misleading for multi-rig polecats
• Token usage (agent.usage, PR #2199) cannot be attributed to a specific bead, rig, or molecule
• bd.call, mail, done events carry no indication of which work item triggered them

Proposed solution (see Roadmap P0):

• At each gt prime, write GT_WORK_RIG, GT_WORK_BEAD, GT_WORK_MOL into the tmux session via SetEnvironment — all subprocesses inherit automatically
• Emit work_rig, work_bead, work_mol on the prime event
• All events emitted after a prime (until the next one) carry the current work context via the inherited env vars

Data Model

See OTel Data Model for complete schema of all events.

The data model is independent of backend — any OTLP-compatible consumer can parse and query these events.

Queries

Metrics (Any OTLP-compatible backend)

These examples use PromQL/MetricsQL syntax, as supported by VictoriaMetrics, Prometheus, Grafana Mimir, etc.

Total counts by status:

sum(rate(gastown_polecat_spawns_total[5m])) by (status)
sum(rate(gastown_bd_calls_total[5m])) by (subcommand, status)

Naming note: OTel SDK metric names use dot notation (e.g. gastown.bd.calls.total). Prometheus-compatible backends export these with underscores (e.g. gastown_bd_calls_total). Use underscore form in PromQL queries.

Latency distributions:

histogram_quantile(0.5, rate(gastown_bd_duration_ms_bucket[5m])) by (subcommand)
histogram_quantile(0.95, rate(gastown_bd_duration_ms_bucket[5m])) by (subcommand)
histogram_quantile(0.99, rate(gastown_bd_duration_ms_bucket[5m])) by (subcommand)

Session activity:

sum(increase(gastown_session_starts_total[1h])) by (role)
sum(increase(gastown_done_total[1h])) by (exit_type)

VictoriaLogs (Structured Logs)

Find all events from a polecat:

gt.agent:Toast

Error analysis:

status:error
_msg:bd.call AND status:error
_msg:session.stop AND status:error

Polecat lifecycle:

_msg:polecat.spawn
_msg:polecat.remove
_msg:agent.state_change AND new_state:working

Debugging Examples

Track a polecat working across multiple rigs:

gt.agent:Toast

Shows all events from polecat Toast, regardless of rig assignment.

Identify sessions with high error rates:

_msg:bd.call AND status:error

Find all events for a run (after PR #2199):

run.id:uuid-1234

Related Documentation

• OTel Data Model — Complete event schema
• Polecat Lifecycle — Persistent polecat model
• Overview — Role taxonomy and architecture
• Reference — Environment variables and commands

Backends Compatible with OTLP

Production Recommendation: For production deployments, consider using the OpenTelemetry Collector as a sidecar. The Collector provides:

• Single agent for all telemetry
• Advanced processing and batching
• Support for multiple backends simultaneously
• Better resource efficiency than per-process exporters

Appendix: Source Reference Audit

Audited against origin/main @ 2d8d71ee35fafda3bbdf353683692bfcc9165476

Initialization (internal/telemetry/telemetry.go)

Subprocess correlation (internal/telemetry/subprocess.go)

Recording (internal/telemetry/recorder.go)

Absent functions and features (confirmed by grep on origin/main)

PromQL naming convention

OTel SDK uses dot notation; Prometheus-compatible backends export with underscores.

PR #2199 additions (commit 8b88de15, not yet on main)