integration_tests/README.md
These tests exercise the exact wheel produced by uv build after installing it into clean virtual environments. The integration_tests/ directory, repository automation metadata, and local dependency/type-checking caches are excluded from published distributions.
Run the complete release-oriented matrix with:
export UV_DEFAULT_INDEX=https://pypi.org/simple
make integration-tests
make integration-tests-release runs the release-safe live matrix and the local Docker security contract in strict mode, so an unavailable daemon, image, credential, or required capability fails the release gate instead of becoming a skip. The focused make integration-tests-security target runs the same wheel and sdist security contract in strict mode without the live provider matrix; the security profile remains separate from the credential-free PR packaging job. make integration-tests-nightly also includes extended capability and transport checks, while make integration-tests-manual includes checks reserved for an intentionally configured manual run. Focused entry points are make integration-tests-packaging, make integration-tests-security, make integration-tests-mcp-v1, make integration-tests-core, make integration-tests-providers, make integration-tests-providers-external, make integration-tests-providers-all, make integration-tests-realtime, make integration-tests-voice, make integration-tests-hosted, and make integration-tests-extras. The packaging profile validates the released public API manifest and historical RunState corpus from base wheel and sdist environments, then validates the public API again from wheel and sdist environments with the Cloudflare extra installed so dependency-conditional exports are required. The security profile installs the Docker extra for both distribution formats, checks packaged credential redaction, and runs model-controlled environment, filesystem, and process inspection inside a local Docker sandbox through the public Runner lifecycle. The MCP v1 profile installs the built wheel with both the supported v1 floor and latest tested v1 release in clean environments; the regular test job validates the locked MCP v2 dependency.
Release PR preparation updates the rolling API manifest locally rather than in a credentialed GitHub workflow. After the release branch version bump, run make update-released-api-contract VERSION=<version>, review and commit the manifest diff, then run make check-released-api-contract VERSION=<version> after subsequent rebases. Promotion fails before writing if the candidate breaks the committed released contract. The prospective release-contract job performs this source validation in one dedicated Python process so provider behavior tests cannot change its import graph. Inspectable top-level classes and functions are promoted automatically; documented properties, intended submodule paths, and canonical aliases remain explicit review decisions recorded in the manifest. The packaged profiles remain the artifact-level verification that the committed contract holds for core and policy-declared optional surfaces across wheel, sdist, and supported platforms.
Invoke the repository-local $integration-tests skill to run the release profile with configured OpenRouter-backed provider checks. OpenRouter provides a single configured gateway for the standard multi-provider matrix; provider-specific direct connections are optional extensions selected explicitly. When a release review also requires runnable examples, run $examples-auto-run first and then $integration-tests.
Set OPENAI_API_KEY for live OpenAI calls. Override OPENAI_AGENTS_INTEGRATION_MODEL, OPENAI_AGENTS_INTEGRATION_REALTIME_MODEL, OPENAI_AGENTS_INTEGRATION_ANY_LLM_MODELS, and OPENAI_AGENTS_INTEGRATION_LITELLM_MODELS when testing different models or configured providers. Provider model lists contain comma-separated adapter model names and require the credentials matching each selected provider. Set OPENAI_AGENTS_INTEGRATION_MCP_SERVER_URL to use another trusted DeepWiki-compatible hosted MCP server that exposes the ask_question tool and can answer questions about the openai/openai-agents-python repository.
Run make integration-tests-providers-external with OPENROUTER_API_KEY to exercise current OpenAI, Anthropic, and Google models through one provider gateway. To extend the matrix with separately configured direct-provider credentials, use make integration-tests-providers-external -- --all, make integration-tests-providers-all, or uv run python .github/scripts/run_integration_tests.py --profile providers --all. Set ANTHROPIC_API_KEY and GEMINI_API_KEY or GOOGLE_API_KEY for the direct providers you want to include. Override OPENAI_AGENTS_INTEGRATION_ANTHROPIC_MODEL, OPENAI_AGENTS_INTEGRATION_GEMINI_MODEL, or the comma-separated OPENAI_AGENTS_INTEGRATION_OPENROUTER_MODELS to select provider models.
The default general model is gpt-5.6, while LiteLLM function-tool cases use the Chat Completions-native openai/gpt-4.1-mini. This avoids LiteLLM's separate Responses API bridge and keeps the adapter regression focused on its actual Chat Completions contract.
When the host requires a SOCKS proxy, the runner installs httpx[socks] as a test-harness dependency without changing the SDK's published requirements. Set OPENAI_AGENTS_INTEGRATION_DISABLE_PROXY=1 when the selected environment should connect without inherited proxy settings.
Set OPENAI_AGENTS_INTEGRATION_STRICT=1 to fail rather than skip when a requested live feature is not configured. The release and security profiles enable strict mode unconditionally. Integration tests never run as part of ordinary make tests.
The security profile requires a reachable local Docker daemon and pulls busybox:1.36.1 by default. Set OPENAI_AGENTS_INTEGRATION_SECURITY_IMAGE to use a pre-approved replacement image. An unavailable daemon or image is a failure for both focused security and release-candidate runs.
Each live test has a 75-second timeout so a stalled provider connection cannot block a release review indefinitely.
Each isolated environment writes a JUnit report to .tmp/integration-tests/results/<profile>/. The runner also prints pass, failure, error, skip, and deselection counts for every profile/environment pair. Pytest output and raw provider payloads are not attached to passing JUnit cases.
Set OPENAI_AGENTS_INTEGRATION_PYTHON to choose the Python interpreter used for isolated environments. For example, OPENAI_AGENTS_INTEGRATION_PYTHON=3.10 make integration-tests-packaging verifies the minimum supported Python package and import boundary; use Python 3.11 or newer for the full adapter matrix because the AnyLLM extra requires Python 3.11.
The release suite also covers canonical and supported legacy public-import identity, client-side handoffs, nested agents as tools, custom and shell tools, namespaced tool search, approval/rejection plus serialized RunState resume, durable SQLite sessions, explicit and server-managed conversation continuation, controlled retries, input/output and tool guardrails, explicit prompt caching, structured streaming output, provider token logprobs, hosted web search/MCP approval, hosted multi-agent streaming, programmatic-tool streaming/handoffs, multi-turn Realtime history, usage, handoffs, agent updates, voice failure propagation, and independent installation of each selected optional dependency group. The nightly profile adds extended approval matrices, parallel tool concurrency, stateless reasoning replay, reusable Responses WebSocket sessions, collected trace trees, streamed provider tool calls, Realtime audio/guardrails, and streamed-input voice pipelines.