Observability And Debugging

OTEL Ownership

Use packages/observability-otel/src/modules/agent-signal/index.ts for the shared tracer and metrics.

Available instruments:

• tracer
• sourceCounter
• signalCounter
• actionCounter
• actionResultCounter
• chainCounter
• signalActionTransitionCounter
• chainDurationHistogram
• actionDurationHistogram
• sourceEventCounter
• sourceEventDurationHistogram
• workflowRunCounter
• workflowRunDurationHistogram
• handlerCounter
• handlerDurationHistogram
• terminalResultCounter

Use this module when you need shared telemetry ownership instead of creating feature-local meters or tracers.

Projection Pipeline

After runtime execution, the service projects one compact observability model from the full chain.

Read:

• apps/server/src/services/agentSignal/observability/projector.ts
• apps/server/src/services/agentSignal/observability/traceEvents.ts
• apps/server/src/services/agentSignal/observability/store.ts

Projection outputs:

• a trace envelope with source, signals, actions, results, edges, and handler runs
• a compact telemetry record with dominant path, status breakdown, and chain metadata

This projection is built from:

• source node
• emitted signals
• planned actions
• executor results

Receipt projection is separate from observability projection. The orchestrator still calls projectAgentSignalReceipts(...) after an immediate runtime chain, but memory and skill self-iteration receipts are projected from agent.execution.completed via buildSelfIterationReceipts(...) and persisted through the receipt store.

How To Inspect A Chain

Use this order:

1. Inspect the source type and payload.
2. Inspect emitted signals.
3. Inspect planned actions.
4. Inspect executor results.
5. Inspect projected edges and dominant path.

The helper toAgentSignalTraceEvents(...) flattens a chain into compact event records suitable for tracing snapshots.

Workflow Snapshot Bridge

Workflow-triggered runs do not naturally pass through the normal foreground runtime snapshot path, so runAgentSignalWorkflow adds a development-only bridge into .agent-tracing/.

Read:

• apps/server/src/workflows/agentSignal/run.ts

Use that path when:

• the source was enqueued with enqueueAgentSignalSourceEvent(...)
• you need local trace visibility for quiet background work

Common Debug Questions

The source emits but nothing happens

Check:

• feature gate enabled for the user
• source type matches a registered source handler
• dedupe or scope lock did not short-circuit generation

Read:

• apps/server/src/services/agentSignal/index.ts
• apps/server/src/services/agentSignal/sources/index.ts

The signal exists but no action runs

Check:

• the signal type has a registered signal handler
• the signal handler returns status: 'dispatch'
• the handler actually returned actions

The action runs twice

Check:

• source dedupe key stability
• action idempotency strategy
• scope key stability across retries and workflow handoff

Reference:

• apps/server/src/services/agentSignal/policies/actionIdempotency.ts
• apps/server/src/services/agentSignal/policies/analyzeIntent/actions/userMemory.ts

Background runs are hard to discover

Check:

• workflow snapshot bridge in development
• projected telemetry record contents
• OTEL counters and histograms in the shared module

Memory or skill receipts are missing after an action applied

Check:

• the action handler enqueued execAgent and stamped an Agent Signal operation marker
• agent.execution.completed carried a selfIteration payload from the run's finalState
• createCompletionPolicy(...) was wired with onSelfIterationCompleted
• buildSelfIterationReceipts(...) recognized the durable mutation api name
• the receipt store accepted the deterministic receipt id
• the marker includes usable anchorMessageId or triggerMessageId when UI placement matters

Minimal Completion Checklist

• source ingress is testable
• handler registration is discoverable from the policy factory
• action executor returns structured results
• projection includes the new path cleanly
• async self-iteration receipts, when relevant, are covered on the completion path
• tests cover at least one happy path and one no-op or failure path