docs/gateway/opentelemetry.md
OpenClaw exports diagnostics through the official diagnostics-otel plugin
using OTLP/HTTP (protobuf). Logs can also be written as stdout JSONL for
container and sandbox log pipelines. Any collector or backend that accepts
OTLP/HTTP works without code changes. For local file logs, see
Logging.
diagnostics-otel subscribes to those events and exports them as
OpenTelemetry metrics, traces, and logs over OTLP/HTTP, and can
mirror log records to stdout JSONL.traceparent header from OpenClaw's
trusted model-call span context when the provider transport accepts custom
headers. Plugin-emitted trace context is not propagated.openclaw plugins install clawhub:@openclaw/diagnostics-otel
{
plugins: {
allow: ["diagnostics-otel"],
entries: {
"diagnostics-otel": { enabled: true },
},
},
diagnostics: {
enabled: true,
otel: {
enabled: true,
endpoint: "http://otel-collector:4318",
protocol: "http/protobuf",
serviceName: "openclaw-gateway",
traces: true,
metrics: true,
logs: true,
sampleRate: 0.2,
flushIntervalMs: 60000,
},
},
}
Or enable the plugin from the CLI: openclaw plugins enable diagnostics-otel.
| Signal | What goes in it |
|---|---|
| Metrics | Counters/histograms for token usage, cost, run duration, failover, skill usage, message flow, Talk events, queue lanes, session state/recovery, tool execution, exec, memory, liveness, and exporter health. |
| Traces | Spans for model usage, model calls, harness lifecycle, skill usage, tool execution, exec, webhook/message processing, context assembly, and tool loops. |
| Logs | Structured logging.file records exported over OTLP or stdout JSONL when diagnostics.otel.logs is enabled; log bodies are withheld unless content capture is explicitly enabled. |
Toggle traces, metrics, and logs independently. Traces and metrics
default to on when diagnostics.otel.enabled is true; logs default to off
and export only when diagnostics.otel.logs is explicitly true. Log export
defaults to OTLP; set diagnostics.otel.logsExporter to stdout for JSONL on
stdout, or both for both.
{
diagnostics: {
enabled: true,
otel: {
enabled: true,
endpoint: "http://otel-collector:4318",
tracesEndpoint: "http://otel-collector:4318/v1/traces",
metricsEndpoint: "http://otel-collector:4318/v1/metrics",
logsEndpoint: "http://otel-collector:4318/v1/logs",
protocol: "http/protobuf", // grpc disables OTLP export
serviceName: "openclaw-gateway", // unset falls back to OTEL_SERVICE_NAME, then "openclaw"
headers: { "x-collector-token": "..." },
traces: true,
metrics: true,
logs: true,
logsExporter: "otlp", // otlp | stdout | both
sampleRate: 0.2, // root-span sampler, 0.0..1.0
flushIntervalMs: 60000, // metric export interval (min 1000ms)
captureContent: {
enabled: false,
inputMessages: false,
outputMessages: false,
toolInputs: false,
toolOutputs: false,
systemPrompt: false,
toolDefinitions: false,
},
},
},
}
| Variable | Purpose |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | Fallback for diagnostics.otel.endpoint when the config key is unset. |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_METRICS_ENDPOINT / OTEL_EXPORTER_OTLP_LOGS_ENDPOINT | Signal-specific endpoint fallbacks used when the matching diagnostics.otel.*Endpoint config key is unset. Signal-specific config wins over signal-specific env, which wins over the shared endpoint. |
OTEL_SERVICE_NAME | Fallback for diagnostics.otel.serviceName when the config key is unset. Default service name is openclaw. |
OTEL_EXPORTER_OTLP_PROTOCOL | Fallback for the wire protocol when diagnostics.otel.protocol is unset. Only http/protobuf enables export. |
OTEL_SEMCONV_STABILITY_OPT_IN | Set to gen_ai_latest_experimental to emit the latest GenAI inference span shape: {gen_ai.operation.name} {gen_ai.request.model} span names, CLIENT span kind, and gen_ai.provider.name instead of the legacy gen_ai.system. GenAI metrics always use bounded, low-cardinality attributes regardless. |
OPENCLAW_OTEL_PRELOADED | Set to 1 when another preload or host process already registered the global OpenTelemetry SDK. The plugin then skips its own NodeSDK lifecycle but still wires diagnostic listeners and honors traces/metrics/logs. |
Raw model/tool content is not exported by default. Spans carry bounded
identifiers (channel, provider, model, error category, hash-only request ids,
tool source, tool owner, skill name/source) and never include prompt text,
response text, tool inputs, tool outputs, skill file paths, or session keys.
Values that look like scoped agent session keys (for example starting with
agent:) are replaced with unknown on low-cardinality attributes. OTLP log
records keep severity, logger, code location, trusted trace context, and
sanitized attributes by default; the raw log message body is exported only
when diagnostics.otel.captureContent is boolean true. Granular
captureContent.* subkeys never enable log bodies. Talk metrics export only
bounded event metadata (mode, transport, provider, event type) - no
transcripts, audio payloads, session ids, turn ids, call ids, room ids, or
handoff tokens.
Outbound model requests may include a W3C traceparent header generated only
from OpenClaw-owned diagnostic trace context for the active model call.
Existing caller-supplied traceparent headers are replaced, so plugins or
custom provider options cannot spoof cross-service trace ancestry.
Set diagnostics.otel.captureContent.* to true only when your collector
and retention policy are approved for prompt, response, tool, or
system-prompt text. Each subkey is independent:
inputMessages - user prompt content.outputMessages - model response content.toolInputs - tool argument payloads.toolOutputs - tool result payloads.systemPrompt - assembled system/developer prompt.toolDefinitions - model tool names, descriptions, and schemas.When any subkey is enabled, model and tool spans get bounded, redacted
openclaw.content.* attributes for that class only.
toolInputs/toolOutputs content is captured for the built-in agent
runtime's tool executions (openclaw.content.tool_input and
gen_ai.tool.call.arguments on completed/error spans;
openclaw.content.tool_output and gen_ai.tool.call.result on completed
spans). The openclaw.content.* names remain the stable OpenClaw attribute
names; the gen_ai.tool.call.* copies mirror them for semconv-native viewers.
External harness tool calls (Codex, Claude CLI) emit
tool.execution.* spans without content payloads. Captured content travels on a
trusted, listener-only channel and is never placed on the public diagnostic event
bus.
diagnostics.otel.sampleRate sets a TraceIdRatioBasedSampler
on the root span only (0.0 drops all, 1.0 keeps all). Unset uses the
OpenTelemetry SDK default (always-on).diagnostics.otel.flushIntervalMs (clamped to a minimum of
1000); unset uses the SDK's periodic-export default.logging.level (file log level) and use the
diagnostic log-record redaction path, not console formatting. High-volume
installs should prefer OTLP collector sampling/filtering over local
sampling. Set diagnostics.otel.logsExporter: "stdout" when your platform
already ships stdout/stderr to a log processor and you have no OTLP logs
collector. Stdout records are one JSON object per line with ts, signal,
service.name, severity, body, redacted attributes, and trusted trace
fields when available.traceId,
spanId, parentSpanId, and traceFlags when the log call carries a valid
diagnostic trace context, letting log processors join local log lines with
exported spans.traceparent headers stay on the
same trace.openclaw.model.call spans include safe prompt
component sizes by default and per-call token attributes when the provider
result exposes usage. openclaw.model.usage remains the run-level
accounting span for aggregate cost, context, and channel dashboards, and
stays on the same diagnostic trace when the emitting runtime has trusted
trace context.openclaw.tokens (counter, attrs: openclaw.token, openclaw.channel, openclaw.provider, openclaw.model, openclaw.agent)openclaw.cost.usd (counter, attrs: openclaw.channel, openclaw.provider, openclaw.model)openclaw.run.duration_ms (histogram, attrs: openclaw.channel, openclaw.provider, openclaw.model)openclaw.context.tokens (histogram, attrs: openclaw.context, openclaw.channel, openclaw.provider, openclaw.model)gen_ai.client.token.usage (histogram, GenAI semantic-conventions metric, attrs: gen_ai.token.type = input/output, gen_ai.provider.name, gen_ai.operation.name, gen_ai.request.model)gen_ai.client.operation.duration (histogram, seconds, GenAI semantic-conventions metric, attrs: gen_ai.provider.name, gen_ai.operation.name, gen_ai.request.model, optional error.type)openclaw.model_call.duration_ms (histogram, attrs: openclaw.provider, openclaw.model, openclaw.api, openclaw.transport, plus openclaw.errorCategory and openclaw.failureKind on classified errors)openclaw.model_call.request_bytes (histogram, UTF-8 byte size of the final model request payload; no raw payload content)openclaw.model_call.response_bytes (histogram, UTF-8 byte size of streamed response chunk payloads; high-frequency text, thinking, and tool-call deltas count only incremental delta bytes; no raw response content)openclaw.model_call.time_to_first_byte_ms (histogram, elapsed time before the first streamed response event)openclaw.model.failover (counter, attrs: openclaw.provider, openclaw.model, openclaw.failover.to_provider, openclaw.failover.to_model, openclaw.failover.reason, openclaw.failover.suspended, openclaw.lane)openclaw.skill.used (counter, attrs: openclaw.skill.name, openclaw.skill.source, openclaw.skill.activation, optional openclaw.agent, optional openclaw.toolName)openclaw.webhook.received (counter, attrs: openclaw.channel, openclaw.webhook)openclaw.webhook.error (counter, attrs: openclaw.channel, openclaw.webhook)openclaw.webhook.duration_ms (histogram, attrs: openclaw.channel, openclaw.webhook)openclaw.message.queued (counter, attrs: openclaw.channel, openclaw.source)openclaw.message.received (counter, attrs: openclaw.channel, openclaw.source)openclaw.message.dispatch.started (counter, attrs: openclaw.channel, openclaw.source)openclaw.message.dispatch.completed (counter, attrs: openclaw.channel, openclaw.outcome, openclaw.reason, openclaw.source)openclaw.message.dispatch.duration_ms (histogram, attrs: openclaw.channel, openclaw.outcome, openclaw.reason, openclaw.source)openclaw.message.processed (counter, attrs: openclaw.channel, openclaw.outcome)openclaw.message.duration_ms (histogram, attrs: openclaw.channel, openclaw.outcome)openclaw.message.delivery.started (counter, attrs: openclaw.channel, openclaw.delivery.kind)openclaw.message.delivery.duration_ms (histogram, attrs: openclaw.channel, openclaw.delivery.kind, openclaw.outcome, openclaw.errorCategory)openclaw.talk.event (counter, attrs: openclaw.talk.event_type, openclaw.talk.mode, openclaw.talk.transport, openclaw.talk.brain, openclaw.talk.provider)openclaw.talk.event.duration_ms (histogram, attrs: same as openclaw.talk.event; emitted when a Talk event reports duration)openclaw.talk.audio.bytes (histogram, attrs: same as openclaw.talk.event; emitted for Talk audio frame events that report byte length)openclaw.queue.lane.enqueue (counter, attrs: openclaw.lane)openclaw.queue.lane.dequeue (counter, attrs: openclaw.lane)openclaw.queue.depth (histogram, attrs: openclaw.lane or openclaw.channel=heartbeat)openclaw.queue.wait_ms (histogram, attrs: openclaw.lane)openclaw.session.state (counter, attrs: openclaw.state, openclaw.reason)openclaw.session.stuck (counter, attrs: openclaw.state; emitted for recoverable stale session bookkeeping)openclaw.session.stuck_age_ms (histogram, attrs: openclaw.state; emitted for recoverable stale session bookkeeping)openclaw.session.turn.created (counter, attrs: openclaw.agent, openclaw.channel, openclaw.trigger)openclaw.session.recovery.requested (counter, attrs: openclaw.state, openclaw.action, openclaw.active_work_kind, openclaw.reason)openclaw.session.recovery.completed (counter, attrs: openclaw.state, openclaw.action, openclaw.status, openclaw.active_work_kind, openclaw.reason)openclaw.session.recovery.age_ms (histogram, attrs: same as the matching recovery counter)openclaw.run.attempt (counter, attrs: openclaw.attempt)diagnostics.stuckSessionWarnMs is the no-progress age threshold for session
liveness diagnostics. A processing session does not age toward this
threshold while OpenClaw observes reply, tool, status, block, or ACP runtime
progress. Typing keepalives do not count as progress, so a silent model or
harness can still be detected.
OpenClaw classifies sessions by the work it can still observe:
session.long_running: active embedded work, model calls, or tool calls
are still making progress. Owned model calls that stay silent past
diagnostics.stuckSessionWarnMs also report as long-running before
diagnostics.stuckSessionAbortMs, so slow or non-streaming model providers
do not look like stalled gateway sessions while abort-observable.session.stalled: active work exists, but the active run has not reported
recent progress. Owned model calls switch from session.long_running to
session.stalled at or after diagnostics.stuckSessionAbortMs; ownerless
stale model/tool activity is not treated as harmless long-running work.
Stalled embedded runs stay observe-only at first, then abort-drain after
diagnostics.stuckSessionAbortMs with no progress so queued turns behind
the lane can resume. When unset, the abort threshold defaults to the safer
extended window of at least 5 minutes and 3x
diagnostics.stuckSessionWarnMs.session.stuck: stale session bookkeeping with no active work, or an idle
queued session with stale ownerless model/tool activity. This releases the
affected session lane immediately after recovery gates pass.Recovery emits structured session.recovery.requested and
session.recovery.completed events. Diagnostic session state is marked idle
only after a mutating recovery outcome (aborted or released) and only if
the same processing generation is still current.
Only session.stuck emits the openclaw.session.stuck counter, the
openclaw.session.stuck_age_ms histogram, and the openclaw.session.stuck
span. Repeated session.stuck diagnostics back off while the session remains
unchanged, so dashboards should alert on sustained increases rather than
every heartbeat tick. For the config knob and defaults, see
Configuration reference.
Liveness warnings also emit:
openclaw.liveness.warning (counter, attrs: openclaw.liveness.reason)openclaw.liveness.event_loop_delay_p99_ms (histogram, attrs: openclaw.liveness.reason)openclaw.liveness.event_loop_delay_max_ms (histogram, attrs: openclaw.liveness.reason)openclaw.liveness.event_loop_utilization (histogram, attrs: openclaw.liveness.reason)openclaw.liveness.cpu_core_ratio (histogram, attrs: openclaw.liveness.reason)openclaw.harness.duration_ms (histogram, attrs: openclaw.harness.id, openclaw.harness.plugin, openclaw.outcome, openclaw.harness.phase on errors)openclaw.tool.execution.duration_ms (histogram, attrs: gen_ai.tool.name, openclaw.toolName, openclaw.tool.source, openclaw.tool.owner, openclaw.tool.params.kind, plus openclaw.errorCategory on errors)openclaw.tool.execution.blocked (counter, attrs: gen_ai.tool.name, openclaw.toolName, openclaw.tool.source, openclaw.tool.owner, openclaw.tool.params.kind, openclaw.deniedReason)openclaw.tool.loop (counter, attrs: openclaw.toolName, openclaw.loop.level, openclaw.loop.action, openclaw.loop.detector, openclaw.loop.count, optional openclaw.loop.paired_tool; emitted when a repetitive tool-call loop is detected)openclaw.exec.duration_ms (histogram, attrs: openclaw.exec.target, openclaw.exec.mode, openclaw.outcome, openclaw.failureKind)openclaw.payload.large (counter, attrs: openclaw.payload.surface, openclaw.payload.action, openclaw.channel, openclaw.plugin, openclaw.reason)openclaw.payload.large_bytes (histogram, attrs: same as openclaw.payload.large)openclaw.memory.rss_bytes / openclaw.memory.heap_used_bytes / openclaw.memory.heap_total_bytes / openclaw.memory.external_bytes / openclaw.memory.array_buffers_bytes (histograms, no attrs; process memory samples)openclaw.memory.pressure (counter, attrs: openclaw.memory.level, openclaw.memory.reason)openclaw.diagnostic.async_queue.dropped (counter, attrs: openclaw.diagnostic.async_queue.drop_class; internal diagnostic-queue backpressure drops)openclaw.telemetry.exporter.events (counter, attrs: openclaw.exporter, openclaw.signal, openclaw.status, optional openclaw.reason, optional openclaw.errorCategory; exporter lifecycle/failure self-telemetry)openclaw.model.usage
openclaw.channel, openclaw.provider, openclaw.modelopenclaw.tokens.* (input/output/cache_read/cache_write/total)gen_ai.system by default, or gen_ai.provider.name when the latest GenAI semantic conventions are opted ingen_ai.request.model, gen_ai.operation.name, gen_ai.usage.*openclaw.run
openclaw.outcome, openclaw.channel, openclaw.provider, openclaw.model, openclaw.errorCategoryopenclaw.model.call
gen_ai.system by default, or gen_ai.provider.name when the latest GenAI semantic conventions are opted ingen_ai.request.model, gen_ai.operation.name, openclaw.provider, openclaw.model, openclaw.api, openclaw.transportopenclaw.errorCategory, error.type, and optional openclaw.failureKind on errorsopenclaw.model_call.request_bytes, openclaw.model_call.response_bytes, openclaw.model_call.time_to_first_byte_msopenclaw.model_call.prompt.input_messages_count, openclaw.model_call.prompt.input_messages_chars, openclaw.model_call.prompt.system_prompt_chars, openclaw.model_call.prompt.tool_definitions_count, openclaw.model_call.prompt.tool_definitions_chars, openclaw.model_call.prompt.total_chars (safe component sizes only, no prompt text)openclaw.model_call.usage.* and gen_ai.usage.* when the model-call result carries provider usage for that individual callopenclaw.provider.request with attribute openclaw.upstreamRequestIdHash (bounded, hash-based) when the upstream provider result exposes a request id; raw ids are never exportedOTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, model-call spans use the latest GenAI inference span name {gen_ai.operation.name} {gen_ai.request.model} and CLIENT span kind instead of openclaw.model.call.openclaw.harness.run
openclaw.harness.id, openclaw.harness.plugin, openclaw.outcome, openclaw.provider, openclaw.model, openclaw.channelopenclaw.harness.result_classification, openclaw.harness.yield_detected, openclaw.harness.items.started, openclaw.harness.items.completed, openclaw.harness.items.activeopenclaw.harness.phase, openclaw.errorCategory, optional openclaw.harness.cleanup_failedopenclaw.tool.execution
gen_ai.tool.name, gen_ai.operation.name (execute_tool), openclaw.toolName, openclaw.tool.source, optional gen_ai.tool.call.id, openclaw.tool.owner, openclaw.tool.params.*openclaw.errorCategory/openclaw.errorCode on errors, openclaw.deniedReason and openclaw.outcome=blocked when denied by policy or sandboxopenclaw.exec
openclaw.exec.target, openclaw.exec.mode, openclaw.outcome, openclaw.failureKind, openclaw.exec.command_length, openclaw.exec.exit_code, openclaw.exec.exit_signal, openclaw.exec.timed_outopenclaw.webhook.processed
openclaw.channel, openclaw.webhookopenclaw.webhook.error
openclaw.channel, openclaw.webhook, openclaw.erroropenclaw.message.processed
openclaw.channel, openclaw.outcome, openclaw.reasonopenclaw.message.delivery
openclaw.channel, openclaw.delivery.kind, openclaw.outcome, openclaw.errorCategory, openclaw.delivery.result_countopenclaw.session.stuck
openclaw.state, openclaw.ageMs, openclaw.queueDepthopenclaw.context.assembled
openclaw.prompt.size, openclaw.history.size, openclaw.context.tokens, openclaw.errorCategory (no prompt, history, response, or session-key content)openclaw.tool.loop
openclaw.toolName, openclaw.loop.level, openclaw.loop.action, openclaw.loop.detector, openclaw.loop.count, optional openclaw.loop.paired_tool (no loop messages, params, or tool output)openclaw.memory.pressure
openclaw.memory.level, openclaw.memory.reason, openclaw.memory.rss_bytes, openclaw.memory.heap_used_bytes, openclaw.memory.heap_total_bytes, openclaw.memory.external_bytes, openclaw.memory.array_buffers_bytes, optional openclaw.memory.threshold_bytes/openclaw.memory.rss_growth_bytes/openclaw.memory.window_msWhen content capture is explicitly enabled, model and tool spans can also
include bounded, redacted openclaw.content.* attributes for the specific
content classes you opted into.
The events below back the metrics and spans above. Plugins can also subscribe to them directly without OTLP export.
Model usage
model.usage - tokens, cost, duration, context, provider/model/channel,
session ids. usage is provider/turn accounting for cost and telemetry;
context.used is the current prompt/context snapshot and can be lower than
provider usage.total when cached input or tool-loop calls are involved.Message flow
webhook.received / webhook.processed / webhook.errormessage.queued / message.processedmessage.delivery.started / message.delivery.completed / message.delivery.errorQueue and session
queue.lane.enqueue / queue.lane.dequeuesession.state / session.long_running / session.stalled / session.stuckrun.attempt / run.progressdiagnostic.heartbeat (aggregate counters: webhooks/queue/session)Harness lifecycle
harness.run.started / harness.run.completed / harness.run.error -
per-run lifecycle for the agent harness. Includes harnessId, optional
pluginId, provider/model/channel, and run id. Completion adds
durationMs, outcome, optional resultClassification, yieldDetected,
and itemLifecycle counts. Errors add phase
(prepare/start/send/resolve/cleanup), errorCategory, and
optional cleanupFailed.Exec
exec.process.completed - terminal outcome, duration, target, mode, exit
code, and failure kind. Command text and working directories are not
included.exec.approval.followup_suppressed - stale approval follow-up dropped
after a session rebound. Includes approvalId, reason
(session_rebound), phase (direct_delivery or gateway_preflight),
and the dispatcher timestamp. Session keys, routes, and command text are
not included.Keep diagnostics events available to plugins or custom sinks without running
diagnostics-otel:
{
diagnostics: { enabled: true },
}
For targeted debug output without raising logging.level, use diagnostics
flags. Flags are case-insensitive and support wildcards (telegram.* or
*):
{
diagnostics: { flags: ["telegram.http"] },
}
Or as a one-off env override:
OPENCLAW_DIAGNOSTICS=telegram.http,telegram.payload openclaw gateway
Flag output goes to the standard log file (logging.file) and is still
redacted by logging.redactSensitive. Full guide:
Diagnostics flags.
{
diagnostics: { otel: { enabled: false } },
}
Or leave diagnostics-otel out of plugins.allow, or run
openclaw plugins disable diagnostics-otel.
diagnostics.* field reference