Agent Tracing CLI Guide

@lobechat/agent-tracing is a zero-config local dev tool that records agent execution snapshots to disk and provides a CLI to inspect them.

How It Works

In NODE_ENV=development, AgentRuntimeService.executeStep() automatically records each step to .agent-tracing/ as partial snapshots. When the operation completes, the partial is finalized into a complete ExecutionSnapshot JSON file.

Data flow: executeStep loop -> build StepPresentationData -> write partial snapshot to disk -> on completion, finalize to .agent-tracing/{timestamp}_{traceId}.json

Context engine capture: In RuntimeExecutors.ts, the call_llm executor calls ctx.tracingContextEngine(input, output) after serverMessagesEngine() processes messages. AgentRuntimeService.executeStep buffers the call per step and forwards it to OperationTraceRecorder.appendStep as the typed contextEngine field. CE flows through this side channel rather than the events array so its heavy payload (agentDocuments, systemRole, …) never enters the Redis state pipeline (LOBE-9110).

Package Location

packages/agent-tracing/
  src/
    types.ts          # ExecutionSnapshot, StepSnapshot, SnapshotSummary
    store/
      types.ts        # ISnapshotStore interface
      file-store.ts   # FileSnapshotStore (.agent-tracing/*.json)
    recorder/
      index.ts        # appendStepToPartial(), finalizeSnapshot()
    viewer/
      index.ts        # Terminal rendering: renderSnapshot, renderStepDetail, renderMessageDetail, renderSummaryTable, renderPayload, renderPayloadTools, renderMemory
    cli/
      index.ts        # CLI entry point (#!/usr/bin/env bun)
      inspect.ts      # Inspect command (default)
      partial.ts      # Partial snapshot commands (list, inspect, clean)
    index.ts          # Barrel exports

Data Storage

• Completed snapshots: .agent-tracing/{ISO-timestamp}_{traceId-short}.json
• Latest symlink: .agent-tracing/latest.json
• In-progress partials: .agent-tracing/_partial/{operationId}.json
• Downloaded remote snapshots: .agent-tracing/_remote/{operationId}.json
• FileSnapshotStore resolves from process.cwd() — run CLI from the repo root

Remote Traces (Production / Staging)

Server deployments also upload completed snapshots to object storage (zstd-compressed; the key is stored in agent_operations.trace_s3_key). agent-tracing inspect <operationId> transparently downloads, decompresses, and caches them — no manual S3 access needed.

1. Find the operation id from a business id (users usually hand you a topic id):

   sqlCopy

   SELECT id, trace_s3_key FROM agent_operations WHERE topic_id = 'tpc_xxx';
   

   trace_s3_key IS NULL means no snapshot was recorded; a non-null key can still 404 in storage (retention/TTL).

2. Configure the base URL — the bucket's public domain plus the /agent-traces prefix — either way:

   • env var: TRACING_BASE_URL=https://<bucket-public-domain>/agent-traces
   • file: .agent-tracing/.env in the repo root containing the same TRACING_BASE_URL=... line

   The deployment-specific value is private to each deployment and intentionally not recorded in this repo.

3. Inspect by operation id — auto-detected by the op_..._agt_..._tpc_... shape; the snapshot is cached to .agent-tracing/_remote/<opId>.json and every inspect flag works the same as for local traces:

   bashCopy

   agent-tracing inspect op_xxx_agt_xxx_tpc_xxx_xxxx    # step tree of a production run
   agent-tracing inspect op_xxx_agt_xxx_tpc_xxx_xxxx -T # tool injection (enabledToolIds, manifests)
   

Implementation: packages/agent-tracing/src/store/remote-store.ts (URL is built from the operation id as {base}/{agentId}/{topicId}/{opId}.json.zst).

CLI Commands

All commands run from the repo root:

# View latest trace (tree overview, `inspect` is the default command)
agent-tracing
agent-tracing inspect
agent-tracing inspect <traceId>
agent-tracing inspect latest

# List recent snapshots
agent-tracing list
agent-tracing list -l 20

# Inspect specific step (-s is short for --step)
agent-tracing inspect <traceId> -s 0

# View messages (-m is short for --messages)
agent-tracing inspect <traceId> -s 0 -m

# View full content of a specific message (by index shown in -m output)
agent-tracing inspect <traceId> -s 0 --msg 2
agent-tracing inspect <traceId> -s 0 --msg-input 1

# View tool call/result details (-t is short for --tools)
agent-tracing inspect <traceId> -s 1 -t

# View raw events (-e is short for --events)
agent-tracing inspect <traceId> -s 0 -e

# View runtime context (-c is short for --context)
agent-tracing inspect <traceId> -s 0 -c

# View context engine input overview (-p is short for --payload)
agent-tracing inspect <traceId> -p
agent-tracing inspect <traceId> -s 0 -p

# View available tools in payload (-T is short for --payload-tools)
agent-tracing inspect <traceId> -T
agent-tracing inspect <traceId> -s 0 -T

# View user memory (-M is short for --memory)
agent-tracing inspect <traceId> -M
agent-tracing inspect <traceId> -s 0 -M

# Raw JSON output (-j is short for --json)
agent-tracing inspect <traceId> -j
agent-tracing inspect <traceId> -s 0 -j

# List in-progress partial snapshots
agent-tracing partial list

# Inspect a partial (use `inspect` directly — all flags work with partial IDs)
agent-tracing inspect <partialOperationId>
agent-tracing inspect <partialOperationId> -T
agent-tracing inspect <partialOperationId> -p

# Clean up stale partial snapshots
agent-tracing partial clean

# Map the context window composition of every LLM call (cm / map are aliases)
agent-tracing ctx-map
agent-tracing ctx-map <operationId|traceId|path.json>
agent-tracing ctx-map --html            # standalone report under .agent-tracing/_reports/
agent-tracing ctx-map --html out.html

ctx-map — Context Window Composition

ctx-map renders one row per call_llm step: the messages that call sent to the model, split into typed segments (system / injected block / user / reasoning / tool call / tool result) with width proportional to tokens, laid against the model's context window.

The second axis is what ctx-lint cannot show: each row is diffed against the previous call, so the longest identical message prefix — the part a provider's prefix cache can reuse — is marked, along with the first message that mutated and the tokens re-processed behind it. A small injected block carrying a relative timestamp (1m ago → now) invalidates every token after it, which shows up as a break marker early in the row.

Reading a row:

• Block colors encode role directly: orange system, green user, blue assistant, gray tool. Assistant reasoning, content, and tool calls use different steps of the same blue scale; framework-injected blocks use a lighter orange than the system prompt. Every value is a step index into a LobeHub scale vendored in viewer/contextMapScales.ts, with assignments in viewer/contextMapPalette.ts. The HTML report ships both themes and follows the system theme.
• Neutral message frames group segments that belong to the same payload message. Every role uses the same frame color: the outline communicates structure only, while the block fill carries role and subtype semantics. The original framed layout uses padding inside each message and a small track gap between messages, keeping each payload message visually distinct.
• Fill says whether the provider reused it: shaded ▓ (HTML: 60%-opaque hatch) was served from the prefix cache; solid █ (HTML: flat) was re-processed by the model.
• The line under the track is the cache ledger — a bracket / green band spanning exactly the cached prefix, then the break marker (▲ in the terminal, a red rule through the track in HTML) at the column where reuse stopped, with the reason and the re-processed tokens.

The track scales to the context window; when the window dwarfs the payloads (a 50k payload on a 1M window) it falls back to the largest call so the composition stays readable, and the header says which basis is in use. Analysis lives in analysis/contextMap.ts and is exported as buildContextMap() for downstream corpus work.

Inspect Flag Reference

Flags marked "Default Step: 0" auto-select step 0 if --step is not provided. All flags support latest or omitted traceId.

Typical Debug Workflow

# 1. Trigger an agent operation in the dev UI

# 2. See the overview
agent-tracing inspect

# 3. List all traces, get traceId
agent-tracing list

# 4. Quick overview of what was fed into context engine
agent-tracing inspect -p

# 5. Inspect a specific step's messages to see what was sent to the LLM
agent-tracing inspect TRACE_ID -s 0 -m

# 6. Drill into a truncated message for full content
agent-tracing inspect TRACE_ID -s 0 --msg 2

# 7. Check available tools vs actual tool calls
agent-tracing inspect -T      # available tools
agent-tracing inspect -s 1 -t # actual tool calls & results

# 8. Inspect user memory injected into the conversation
agent-tracing inspect -M

# 9. Diff system role between steps (multi-step agents)
agent-tracing inspect TRACE_ID -r -d 2

Key Types

interface ExecutionSnapshot {
  traceId: string;
  operationId: string;
  model?: string;
  provider?: string;
  startedAt: number;
  completedAt?: number;
  completionReason?:
    'done' | 'error' | 'interrupted' | 'max_steps' | 'cost_limit' | 'waiting_for_human';
  totalSteps: number;
  totalTokens: number;
  totalCost: number;
  error?: { type: string; message: string };
  steps: StepSnapshot[];
}

interface StepSnapshot {
  stepIndex: number;
  stepType: 'call_llm' | 'call_tool';
  executionTimeMs: number;
  content?: string; // LLM output
  reasoning?: string; // Reasoning/thinking
  inputTokens?: number;
  outputTokens?: number;
  toolsCalling?: Array<{ apiName: string; identifier: string; arguments?: string }>;
  toolsResult?: Array<{
    apiName: string;
    identifier: string;
    isSuccess?: boolean;
    output?: string;
  }>;
  messages?: any[]; // DB messages before step
  context?: { phase: string; payload?: unknown; stepContext?: unknown };
  events?: Array<{ type: string; [key: string]: unknown }>;
  contextEngine?: {
    input?: unknown; // contextEngineInput minus messages + toolsConfig (reconstructible from baseline)
    output?: unknown; // processed messages array (final LLM payload)
  };
}

--messages Output Structure

When using --messages, the output shows three sections (if context engine data is available):

1. Context Engine Input — DB messages passed to the engine, with [0], [1], ... indices. Use --msg-input N to view full content.
2. Context Engine Params — systemRole, model, provider, knowledge, tools, userMemory, etc.
3. Final LLM Payload — Processed messages after context engine (system date injection, user memory, history truncation, etc.), with [0], [1], ... indices. Use --msg N to view full content.

Integration Points

• Recording: apps/server/src/services/agentRuntime/AgentRuntimeService.ts — in the executeStep() method, after building stepPresentationData, writes partial snapshot in dev mode
• Context engine capture: apps/server/src/modules/AgentRuntime/RuntimeExecutors.ts — in call_llm executor, after serverMessagesEngine() returns, calls ctx.tracingContextEngine(input, output). AgentRuntimeService.executeStep buffers it per step and passes it to traceRecorder.appendStep as the typed contextEngine field (kept off the events array to stay out of Redis state).
• Store: FileSnapshotStore reads/writes to .agent-tracing/ relative to process.cwd()