ContextQMD
Back to Dd Trace Js

LLM Observability Testing Skill

.agents/skills/llmobs-testing/SKILL.md

6.9.07.6 KB
Original Source

LLM Observability Testing Skill

Decide how each instrumented surface gets its responses first

That choice picks the response source and test setup — cassettes for a workflow record nothing, while pure-function tests for a provider-backed call miss the network surface entirely. The operation independently determines its span kind and fields. These are working categories for reasoning; none exists as a code constant.

  • LLM client / multi-provider — reaches provider HTTP directly or through a supplied provider package (openai, anthropic, genai, ai, langchain): VCR cassettes or a canned fetch.
  • Orchestration — carries workflow or graph state and makes no provider calls of its own (langgraph): no VCR; drive nodes with plain return values.
  • Infrastructure — implements a protocol or server (modelcontextprotocol-sdk): run the SDK's own server and client over its in-memory transport.
  • Canned fetch instead of a cassette — where the spec supplies the responses itself: google-cloud-vertexai swaps global.fetch per test and stubs Google auth, openai-agents and some ai providers pass a fetch option to the client they construct.

See references/category-strategies.md for the forbidden-vs-required matrix per strategy.

Core Testing Concepts

1. Test Structure

LLMObs tests use special helpers to validate span events.

Key components:

  • useLlmObs() - Initializes LLMObs test environment
  • getEvents() - Retrieves captured span events
  • assertLlmObsSpanEvent() - Validates span structure with flexible matchers

Basic test flow:

  1. Initialize test environment with useLlmObs({ plugin: 'name' })
  2. Call instrumented method (chat completion, workflow execution, etc.)
  3. Get captured span events with getEvents()
  4. Validate span structure with assertLlmObsSpanEvent()

See references/test-structure.md for complete test file templates.

2. VCR Cassettes

Provider traffic is recorded once and replayed afterwards. Clients reach the proxy at http://127.0.0.1:9126/vcr/{provider}; the category block above decides which categories use it at all.

Two facts block every first run:

  • The proxy is the test-agent container, not a script in this repo — docker compose up -d testagent. Without it every call fails with ECONNREFUSED 127.0.0.1:9126, which reads like a provider outage.
  • Cassettes live in one shared tree under packages/dd-trace/test/llmobs/cassettes/{provider}/, with generated names, rather than beside the spec.

See references/vcr-cassettes.md for recording, provider mapping, body normalizers, and the commands to run a single integration.

3. Response Strategy And Operation Kind

The block at the top maps response source to test strategy. The operation maps independently to a span kind:

  • Provider-backed LLM client / multi-provider operations: use the proxy baseURL http://127.0.0.1:9126/vcr/{provider} or a canned fetch. Chat and generation emit llm; LangChain and ai also expose operations with other kinds.
  • Orchestration: span kind 'workflow' or 'agent', never 'llm' — the orchestrator coordinates libraries that call providers rather than calling them itself. Nodes return plain values, so the test exercises graph execution instead of a provider API.
  • Infrastructure: the SDK's own server and client over its in-memory transport, protocol-specific validation, no VCR.

See references/category-strategies.md for the patterns per shape.

4. Assertion Patterns

assertLlmObsSpanEvent(actual, expected)

Validates span structure with flexible matchers for non-deterministic values.

Available matchers: each one is a typeof or nullish check, not a value check.

  • MOCK_STRING - any string, '' included (use for output text)
  • MOCK_NOT_NULLISH - anything but null / undefined, so 0 and '' pass (use for token counts)
  • MOCK_NUMBER - any number
  • MOCK_OBJECT - anything with typeof 'object', null included (opaque schema / metadata payloads, or a whole output message whose shape varies, as the ai specs do)

Required fields: span, spanKind, name, tags. A missing tags throws TypeError: Cannot read properties of undefined (reading 'ml_app') instead of failing an assertion, and every plugin span carries at least { ml_app: 'test', integration: '<integration>' }.

Optional fields: modelName, modelProvider, inputMessages, outputMessages, inputDocuments, outputDocuments, inputValue, outputValue, metrics, metadata, toolDefinitions, error, parentId, sessionId, traceId. Omitting a field asserts its absence rather than ignoring it: no model fields, no input, no output, no metadata, no tool definitions, metrics of {}, status: 'ok', and the root parent id. traceId is the exception: omission defaults to MOCK_STRING because every event has one. See references/assertion-helpers.md for the patterns.

Test File Organization

Location: packages/dd-trace/test/llmobs/plugins/{integration}/index.spec.js. One file per major-version surface when the SDK's shape changed across majors, named after it rather than kept in one file — openaiv3.spec.js / openaiv4.spec.js, index.spec.js / index.v7.spec.js.

Structure:

  1. Import helpers from '../../util'
  2. Initialize LLMObs test environment
  3. Load modules after useLlmObs() installs the tracer, then recreate mutable clients per test
  4. Group tests by method (describe('chat completions', ...))
  5. Cover all instrumented methods
  6. Test error cases

Standard imports:

useLlmObs, assertLlmObsSpanEvent, MOCK_STRING, MOCK_NOT_NULLISH, MOCK_NUMBER, MOCK_OBJECT

See references/test-structure.md for complete template.

Span Kinds And The Fields They Carry

SPAN_KINDS in packages/dd-trace/src/llmobs/constants/tags.js is the list the public SDK validates against: llm (chat / completions), workflow, agent, task (a unit of work inside a workflow), tool, embedding, retrieval. Plugins set the kind directly and skip that validation, so kinds outside the list exist — ai v7 and claude-agent-sdk both emit step.

Pinning a field the kind never emits asserts metadata production does not produce:

  • llm — modelName, modelProvider, inputMessages / outputMessages, and any emitted token metrics / metadata
  • embedding — modelName, modelProvider, inputDocuments, outputValue, sometimes metrics
  • retrieval — inputValue, outputDocuments
  • workflow / agent / task / step / tool — kind-specific inputValue / outputValue, sometimes metadata, never model fields or token metrics

Cover every instrumented method, and a multi-turn conversation where the surface takes one.

Error Handling

On errors the span is still submitted. Match the plugin's output contract: OpenAI and GenAI carry outputMessages: [{ content: '', role: '' }], while Anthropic and non-llm integrations may omit output. Pass a truthy marker to expect an error:

javascript
error: {},

The option decides only whether the expected event carries status: 'error' — assertLlmObsSpanEvent copies the three error fields out of the span it is checking, so the marker does not pin the throw. A call that resolves still fails on that status. Pin which error was thrown on the APM span the LLMObs span was built from:

javascript
assert.strictEqual(apmSpans[0].meta['error.message'], error.message)