integration-opentelemetry/built-in (OpenTelemetry Built-in Tracing)

You can run this example with:

npx promptfoo@latest init --example integration-opentelemetry/built-in
cd integration-opentelemetry/built-in

This example demonstrates promptfoo's built-in OpenTelemetry tracing for LLM provider calls.

Quick Start

1. Set up environment variables:

# Required for the providers you want to test
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
# Add other provider keys as needed

2. Run the evaluation:

npx promptfoo eval -c promptfooconfig.yaml

3. View traces in the UI:

npx promptfoo view

Navigate to the Traces tab to see detailed span information.

Configuration

Tracing is enabled by default. Configure via environment variables:

Viewing Traces Externally

With Jaeger

1. Start Jaeger:

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

2. Run eval with OTLP export:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npx promptfoo eval

3. View at http://localhost:16686

With Honeycomb

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io \
OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY" \
npx promptfoo eval

Trace Attributes

Each LLM call span includes:

GenAI Semantic Conventions

• gen_ai.system - Provider system (openai, anthropic, etc.)
• gen_ai.operation.name - Operation type (chat, completion, embedding)
• gen_ai.request.model - Requested model name
• gen_ai.request.max_tokens - Max tokens setting
• gen_ai.request.temperature - Temperature setting
• gen_ai.usage.input_tokens - Prompt tokens used
• gen_ai.usage.output_tokens - Completion tokens used
• gen_ai.usage.total_tokens - Total tokens
• gen_ai.usage.cached_tokens - Cached tokens (Anthropic)
• gen_ai.usage.reasoning_tokens - Reasoning tokens (o1 models)
• gen_ai.response.model - Actual model used
• gen_ai.response.id - Provider response ID
• gen_ai.response.finish_reasons - Finish reasons

Promptfoo Attributes

• promptfoo.provider.id - Provider identifier
• promptfoo.eval.id - Evaluation run ID
• promptfoo.test.index - Test case index
• promptfoo.prompt.label - Prompt label

Supported Providers

All major providers are instrumented: