When a request enters Plano, it creates an **inbound span** (``plano(inbound)``) that represents the initial request reception and processing. This span captures:

- HTTP request details (method, path, headers)
- Request payload size
- Initial validation and authentication

Orchestration & Routing
~~~~~~~~~~~~~~~~~~~~~~~~

For agent systems, Plano performs intelligent routing through orchestration spans:

- **Agent Orchestration** (``plano(orchestrator)``): When multiple agents are available, Plano uses an LLM to analyze the user's intent and select the most appropriate agent. This span captures the orchestration decision-making process.

- **LLM Routing** (``plano(routing)``): For direct LLM requests, Plano determines the optimal endpoint based on your routing strategy (round-robin, least-latency, cost-optimized). This span includes:

  - Routing strategy used
  - Selected upstream endpoint
  - Route determination time
  - Fallback indicators (if applicable)

Agent Processing
~~~~~~~~~~~~~~~~

When requests are routed to agents, Plano creates spans for agent execution:

- **Agent Filter Chains** (``plano(filter)``): If filters are configured (guardrails, context enrichment, query rewriting), each filter execution is captured in its own span, showing the transformation pipeline.

- **Agent Execution** (``plano(agent)``): The main agent processing span that captures the agent's work, including any tools invoked and intermediate reasoning steps.

Outbound LLM Calls
~~~~~~~~~~~~~~~~~~

All LLM calls—whether from Plano's routing layer or from agents—are traced with **LLM spans** (``plano(llm)``) that capture:

- Model name and provider (e.g., ``gpt-4``, ``claude-3-sonnet``)
- Request parameters (temperature, max_tokens, top_p)
- Token usage (prompt_tokens, completion_tokens)
- Streaming indicators and time-to-first-token
- Response metadata

**Example Span Attributes**::

    # LLM call span
    llm.model = "gpt-4"
    llm.provider = "openai"
    llm.usage.prompt_tokens = 150
    llm.usage.completion_tokens = 75
    llm.duration_ms = 1250
    llm.time_to_first_token = 320

Handoff to Upstream Services

# Span name: "POST /v1/chat/completions gpt-4 🚩"
# Standard LLM attributes:
llm.model = "gpt-4"
llm.usage.total_tokens = 225

# Top-level signal attributes:
signals.quality            = "severe"
signals.quality_score      = 0.0
signals.turn_count         = 15
signals.efficiency_score   = 0.234

# Layered attributes (only non-zero categories are emitted):
signals.interaction.misalignment.count     = 4
signals.interaction.misalignment.severity  = 2
signals.interaction.disengagement.count    = 5
signals.interaction.disengagement.severity = 3

# Per-instance span event:
event: signal.interaction.disengagement.escalation
  signal.type          = "interaction.disengagement.escalation"
  signal.message_index = 14
  signal.confidence    = 1.0
  signal.snippet       = "get me a human"

You configure one or more header prefixes. Any incoming HTTP header whose name starts with one of these
prefixes is captured as a span attribute. You can also provide static attributes that are always injected.

- The **prefix is only for matching**, not the resulting attribute key.
- The attribute key is the header name **with the prefix removed**, then hyphens converted to dots.

.. note::

   Custom span attributes are attached to LLM spans when handling ``/v1/...`` requests via ``llm_chat``. For orchestrator requests to ``/agents/...``,
   these attributes are added to both the orchestrator selection span and to each agent span created by ``agent_chat``.

**Example**

Configured prefix::

  tracing:
    span_attributes:
      header_prefixes:
        - x-katanemo-

Incoming headers::

  X-Katanemo-Workspace-Id: ws_123
  X-Katanemo-Tenant-Id: ten_456

Resulting span attributes::

  workspace.id = "ws_123"
  tenant.id = "ten_456"

Configuration

- **Prefix must match exactly**: ``katanemo-`` does not match ``x-katanemo-`` headers.
- **Trailing dash is recommended**: Without it, ``x-katanemo`` would also match ``x-katanemo-foo`` and
  ``x-katanemofoo``.
- **Keys are always strings**: Values are captured as string attributes.

**Prefix mismatch example**

Config::

  tracing:
    span_attributes:
      header_prefixes:
        - x-katanemo-

Request headers::

  X-Other-User-Id: usr_999

Result: no attributes are captured from ``X-Other-User-Id``.


Benefits of Using ``Traceparent`` Headers
-----------------------------------------

- **Standardization**: The W3C Trace Context standard ensures compatibility across ecosystem tools, allowing
  traces to be propagated uniformly through different layers of the system.
- **Ease of Integration**: OpenTelemetry's design allows developers to easily integrate tracing with minimal
  changes to their codebase, enabling quick adoption of end-to-end observability.
- **Interoperability**: Works seamlessly with popular tracing tools like AWS X-Ray, Datadog, Jaeger, and many others,
  making it easy to visualize traces in the tools you're already usi

How to Initiate A Trace
-----------------------

1. **Enable Tracing Configuration**: Simply add the ``random_sampling`` in ``tracing`` section to 100`` flag to in the :ref:`listener <plano_overview_listeners>` config

2. **Trace Context Propagation**: Plano automatically propagates the ``traceparent`` header. When a request is received, Plano will:

   - Generate a new ``traceparent`` header if one is not present.
   - Extract the trace context from the ``traceparent`` header if it exists.
   - Start a new span representing its processing of the request.
   - Forward the ``traceparent`` header to downstream services.

3. **Sampling Policy**: The 100 in ``random_sampling: 100`` means that all the requests as sampled for tracing.
   You can adjust this value from 0-100.


Tracing with the CLI
--------------------

The Plano CLI ships with a local OTLP/gRPC listener and a trace viewer so you can inspect spans without wiring a full observability backend. This is ideal for development, debugging, and quick QA.

Quick Start
~~~~~~~~~~~

You can enable tracing in either of these ways:

1. Start the local listener explicitly:

.. code-block:: console

  $ planoai trace listen

2. Or start Plano with tracing enabled (auto-starts the local OTLP listener):

.. code-block:: console

  $ planoai up --with-tracing

  # Optional: choose a different listener port
  $ planoai up --with-tracing --tracing-port 4318

3. Send requests through Plano as usual. The listener accepts OTLP/gRPC on:

   - ``0.0.0.0:4317`` (default)

4. View the most recent trace:

.. code-block:: console

  $ planoai trace

Inspect and Filter Traces

- ``--where`` accepts repeatable ``key=value`` filters and uses AND semantics.
- ``--filter`` supports wildcards (``*``) to limit displayed attributes.
- ``--no-interactive`` disables prompts when listing traces.
- By default, inbound/outbound spans use a compact attribute view.


Trace Propagation
-----------------

Plano uses the W3C Trace Context standard for trace propagation, which relies on the ``traceparent`` header.
This header carries tracing information in a standardized format, enabling interoperability between different
tracing systems.

Header Format

To integrate AI tracing, your application needs to follow a few simple steps. The steps
below are very common practice, and not unique to Plano, when you reading tracing headers and export
`spans <https://docs.lightstep.com/docs/understand-distributed-tracing>`_ for distributed tracing.

- Read the ``traceparent`` header from incoming requests.
- Start new spans as children of the extracted context.
- Include the ``traceparent`` header in outbound requests to propagate trace context.
- Send tracing data to a collector or tracing backend to export spans

Example with OpenTelemetry in Python
************************************

Install OpenTelemetry packages:

.. code-block:: console

    $ pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
    $ pip install opentelemetry-instrumentation-requests

Set up the tracer and exporter:

.. code-block:: python

   from opentelemetry import trace
   from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
   from opentelemetry.instrumentation.requests import RequestsInstrumentor
   from opentelemetry.sdk.resources import Resource
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor

   # Define the service name
   resource = Resource(attributes={
       "service.name": "customer-support-agent"
   })

   # Set up the tracer provider and exporter
   tracer_provider = TracerProvider(resource=resource)
   otlp_exporter = OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)
   span_processor = BatchSpanProcessor(otlp_exporter)
   tracer_provider.add_span_processor(span_processor)
   trace.set_tracer_provider(tracer_provider)

   # Instrument HTTP requests
   RequestsInstrumentor().instrument()

Handle incoming requests:

.. code-block:: python

   from opentelemetry import trace
   from opentelemetry.propagate import extract, inject
   import requests

   def handle_request(request):
       # Extract the trace context
       context = extract(request.headers)
       tracer = trace.get_tracer(__name__)

       with tracer.start_as_current_span("process_customer_request", context=context):
           # Example of processing a customer request
           print("Processing customer request...")

           # Prepare headers for outgoing request to payment service
           headers = {}
           inject(headers)

           # Make outgoing request to external service (e.g., payment gateway)
           response = requests.get("http://payment-service/api", headers=headers)

           print(f"Payment service response: {response.content}")


Integrating with Tracing Tools
------------------------------

AWS X-Ray
~~~~~~~~~

To send tracing data to `AWS X-Ray <https://aws.amazon.com/xray/>`_ :

1. **Configure OpenTelemetry Collector**: Set up the collector to export traces to AWS X-Ray.

   Collector configuration (``otel-collector-config.yaml``):

   .. code-block:: yaml

      receivers:
        otlp:
          protocols:
            grpc:

      processors:
        batch:

      exporters:
        awsxray:
          region: <Your-Aws-Region>

      service:
        pipelines:
          traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [awsxray]

2. **Deploy the Collector**: Run the collector as a Docker container, Kubernetes pod, or standalone service.
3. **Ensure AWS Credentials**: Provide AWS credentials to the collector, preferably via IAM roles.
4. **Verify Traces**: Access the AWS X-Ray console to view your traces.

Datadog
~~~~~~~

Datadog

To send tracing data to `Datadog <https://docs.datadoghq.com/getting_started/tracing/>`_:

1. **Configure OpenTelemetry Collector**: Set up the collector to export traces to Datadog.

   Collector configuration (``otel-collector-config.yaml``):

   .. code-block:: yaml

      receivers:
        otlp:
          protocols:
            grpc:

      processors:
        batch:

      exporters:
        datadog:
          api:
            key: "${<Your-Datadog-Api-Key>}"
          site: "${DD_SITE}"

      service:
        pipelines:
          traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [datadog]

2. **Set Environment Variables**: Provide your Datadog API key and site.

   .. code-block:: console

        $ export <Your-Datadog-Api-Key>=<Your-Datadog-Api-Key>
        $ export DD_SITE=datadoghq.com  # Or datadoghq.eu

3. **Deploy the Collector**: Run the collector in your environment.
4. **Verify Traces**: Access the Datadog APM dashboard to view your traces.

Langtrace
~~~~~~~~~

Langtrace is an observability tool designed specifically for large language models (LLMs). It helps you capture, analyze, and understand how LLMs are used in your applications including those built using Plano.

To send tracing data to `Langtrace <https://docs.langtrace.ai/supported-integrations/llm-tools/arch>`_:

1. **Configure Plano**: Make sure Plano is installed and setup correctly. For more information, refer to the `installation guide <https://github.com/katanemo/archgw?tab=readme-ov-file#prerequisites>`_.

2. **Install Langtrace**: Install the Langtrace SDK.:

   .. code-block:: console

        $ pip install langtrace-python-sdk

3. **Set Environment Variables**: Provide your Langtrace API key.

   .. code-block:: console

        $ export LANGTRACE_API_KEY=<Your-Langtrace-Api-Key>

4. **Trace Requests**: Once you have Langtrace set up, you can start tracing requests.

   Here's an example of how to trace a request using the Langtrace Python SDK:

   .. code-block:: python

      import os
      from langtrace_python_sdk import langtrace  # Must precede any llm module imports
      from openai import OpenAI

      langtrace.init(api_key=os.environ['LANGTRACE_API_KEY'])

      client = OpenAI(api_key=os.environ['OPENAI_API_KEY'], base_url="http://localhost:12000/v1")

      response = client.chat.completions.create(
          model="gpt-4o-mini",
          messages=[
              {"role": "system", "content": "You are a helpful assistant"},
              {"role": "user", "content": "Hello"},
          ]
      )

      print(chat_completion.choices[0].message.content)

5. **Verify Traces**: Access the Langtrace dashboard to view your traces.


Best Practices
--------------

- **Consistent Instrumentation**: Ensure all services propagate the ``traceparent`` header.
- **Secure Configuration**: Protect sensitive data and secure communication between services.
- **Performance Monitoring**: Be mindful of the performance impact and adjust sampling rates accordingly.
- **Error Handling**: Implement proper error handling to prevent tracing issues from affecting your application.

Summary
----------

By leveraging the ``traceparent`` header for trace context propagation, Plano enables developers to implement
tracing efficiently. This approach simplifies the process of collecting and analyzing tracing data in common
tools like AWS X-Ray and Datadog, enhancing observability and facilitating faster debugging and optimization.

Additional Resources
--------------------

For full command documentation (including ``planoai trace`` and all other CLI commands), see :ref:`cli_reference`.

External References