.. -- coding: utf-8 --

======== Signals™

Agentic Signals are lightweight, model-free behavioral indicators computed from live interaction trajectories and attached to your existing OpenTelemetry traces. They are the instrumentation layer of a closed-loop improvement flywheel for agents — turning raw production traffic into prioritized data that can drive prompt, routing, and model updates without running an LLM-as-judge on every session.

The framework implemented here follows the taxonomy and detector design in Signals: Trajectory Sampling and Triage for Agentic Interactions (Chen et al., 2026 <https://arxiv.org/abs/2604.00356>_). All detectors are computed without model calls; the entire pipeline attaches structured attributes and span events to existing spans so your dashboards and alerts work unmodified.

Why Signals Matter: The Improvement Flywheel

Agentic applications are increasingly deployed at scale, yet improving them after deployment remains difficult. Production trajectories are long, numerous, and non-deterministic, making exhaustive human review infeasible and auxiliary LLM evaluation expensive. As a result, teams face a bottleneck: they cannot score every response, inspect every trace, or reliably identify which failures and successes should inform the next model update. Without a low-cost triage layer, the feedback loop from production behavior to model improvement remains incomplete.

Signals close this loop by cheaply identifying which interactions among millions are worth inspecting:

Instrument. Live trajectories are scored with model-free signals attached as structured attributes on existing OpenTelemetry spans, organized under a fixed taxonomy of interaction, execution, and environment signals. This requires no additional model calls, infrastructure, or changes to online agent behavior.
Sample & triage. Signal attributes act as filters: they surface severe failures, retrieve representative exemplars, and exclude the uninformative middle. In our experiments, signal-based sampling achieves 82% informativeness on :math:\tau-bench, compared with 54% for random sampling, yielding a 1.52× efficiency gain per informative trajectory.
Data Construction. The triaged subset becomes targeted input for constructing preference datasets or supervised fine-tuning datasets from production trajectories.
Model Optimization. The resulting preference or supervised fine-tuning data is used to update the model through methods such as DPO, RLHF, or supervised fine-tuning, so optimization is driven by targeted production behavior rather than undifferentiated trace noise.
Deploy. The improved model is deployed and immediately re-instrumented with the same signals, enabling teams to measure whether the change improved production behavior and to feed the next iteration.

This loop depends on the first step being nearly free. The framework is therefore designed around fixed-taxonomy, model-free detectors with :math:O(\text{messages}) cost, no online behavior change, and no dependence on expensive evaluator models. By making production traces searchable and sampleable at scale, signals turn raw agent telemetry into a practical model-optimization flywheel.

What Are Behavioral Signals?

Behavioral signals are canaries in the coal mine — early, objective indicators that something may have gone wrong (or gone exceptionally well). They don't explain why an agent failed, but they reliably signal where attention is needed.

These signals emerge naturally from the rhythm of interaction:

A user rephrasing or correcting the same request
Sharp increases in conversation length
Negative stance markers ("this doesn't work", ALL CAPS, excessive !!! or ???)
Agent repetition or tool-call loops
Expressions of gratitude, confirmation, or task success
Requests for a human agent or explicit quit intent
Tool errors, timeouts, rate limits, and context-window exhaustion

Individually, these clues are shallow; together, they form a fingerprint of agent performance. Embedded directly into traces, they make it easy to spot friction as it happens: where users struggle, where agents loop, where tool failures cluster, and where escalations occur.

Signal Taxonomy

Signals are organized into three top-level layers, each with its own intent. Every detected signal belongs to exactly one leaf type under one of seven categories. The per-category summaries and leaf-type descriptions below are borrowed verbatim from the reference implementation at katanemo/signals <https://github.com/katanemo/signals>_ to keep the documentation and the detector contract in sync.

Interaction — user ↔ agent conversational quality

Misalignment — Misalignment signals capture semantic or intent mismatch between the user and the agent, such as rephrasing, corrections, clarifications, and restated constraints. These signals do not assert that either party is "wrong"; they only indicate that shared understanding has not yet been established.