Provider conformance

This harness keeps the services → agents → workflows lifecycle honest across the supported AI providers. It runs the same end-to-end scenarios against each configured provider and treats missing provider keys as an explicit skip, so the suite is safe for local development, forks, and scheduled CI.

What it exercises

go run ./internal/harness/provider-conformance fans out over the harnesses in internal/harness:

• universe — service discovery plus agent tool calls over the real runtime.
• agent-flow — a workflow event that drives an agent to call services.
• plan-delegate — plan persistence plus agent-to-agent delegation and service calls.

The command also emits the registered provider capability matrix so the run shows which providers advertise model, image, video, and streaming support.

Local usage

Run the deterministic path with no secrets:

go run ./internal/harness/provider-conformance -providers mock

Run every live provider that has a key in the environment:

go run ./internal/harness/provider-conformance \
  -summary-json provider-conformance-summary.json \
  -summary-markdown provider-conformance-summary.md \
  -capabilities-markdown provider-capabilities.md

Provider keys are read from MICRO_AI_API_KEY or the provider-specific variable:

Use -require-configured when you want a selected provider without a key to fail instead of skip:

go run ./internal/harness/provider-conformance \
  -providers anthropic,openai \
  -require-configured

Scheduled CI behavior

The Harness (E2E) workflow runs on pushes and pull requests with deterministic mock LLMs. On the daily schedule and manual dispatch it also runs the live provider conformance job. That job:

1. reads the provider keys from repository secrets,
2. skips providers whose secrets are absent,
3. fails when any configured provider fails a harness, and
4. uploads JSON and Markdown coverage artifacts for the run.

The job also appends the Markdown summary and capability matrix to the GitHub Actions step summary, making configured, skipped, and failed provider coverage visible without downloading artifacts.

Adding a provider

To bring a new provider into scheduled conformance:

1. register its ai provider implementation and capability metadata,
2. add the provider name and key variable to providerEnv in main.go,
3. import the provider package in main.go,
4. pass the matching repository secret through .github/workflows/harness.yml, and
5. run go run ./internal/harness/provider-conformance -providers <name> \ -require-configured with a live key before opening the change.