docs/realtime/xai.md
[XaiRealtimeModel][pydantic_ai.realtime.xai.XaiRealtimeModel] brings Grok Voice into the typed,
server-side realtime agent loop. Start with the realtime quickstart or the
text-to-audio example.
To use Grok Voice, install pydantic-ai-slim with the xai-realtime optional group. Alongside
xai-sdk, the bundle includes the openai package, because Grok Voice's realtime API reuses the
OpenAI Realtime protocol's event types:
pip/uv-add "pydantic-ai-slim[xai-realtime]"
Set XAI_API_KEY as described in the xAI model documentation.
Use provider='xai' or pass an
[XaiProvider][pydantic_ai.providers.xai.XaiProvider] with api_key=. Custom api_host is
unsupported, and a provider constructed with only xai_client= cannot open the WebSocket because
the connection requires the API key.
Use a Grok Voice ID such as grok-voice-latest or a pinned grok-voice-think-* model.
grok-voice-latest follows xAI's current flagship and can change underneath an application; pin a
version when behavior must remain stable. Use the
official xAI voice documentation for the canonical
model list.
[XaiRealtimeModelSettings][pydantic_ai.realtime.xai.XaiRealtimeModelSettings] — the realtime
counterpart of model run settings — extends the
shared settings:
from pydantic_ai.realtime.xai import XaiRealtimeModel, XaiRealtimeModelSettings
settings = XaiRealtimeModelSettings(
xai_voice='eve',
turn_detection={'sensitivity': 'low'},
input_transcription_model='auto',
)
model = XaiRealtimeModel('grok-voice-latest', settings=settings)
xai_voice selects the provider voice; when unset, xAI picks its own server-side default
(currently eve). For exact server-VAD threshold or
automatic-response behavior, set xai_turn_detection= with
[ServerVAD][pydantic_ai.realtime.openai.ServerVAD]; it fully overrides shared
turn_detection.
Set turn_detection=False for push-to-talk.
Input transcription defaults to 'auto'. Unlike the incremental
deltas described in live captions, xAI sends cumulative transcript
snapshots that can revise earlier words, so caption UIs should render the full
[TranscriptUpdate.transcript][pydantic_ai.realtime.TranscriptUpdate.transcript] rather than
append deltas.
grok-voice-latest and grok-voice-think-* models support the shared
thinking setting. The provider exposes
only 'high' and 'none': every enabled effort maps to 'high', while False maps to 'none'.
Other Grok Voice models ignore the setting.
| Feature | Support | Notes |
|---|---|---|
| Audio format | Full feature support | Mono PCM16, 24 kHz input and output |
| Text output | Unsupported | Grok Voice always produces audio |
| Image input | Unsupported | Audio/text input only |
| Manual turns | Full feature support | turn_detection=False plus commit/create verbs |
| Interruption | Limited parameter support | interrupt() works; output truncation with played_ms does not |
| Input transcription | Full feature support | Dedicated provider path; 'auto' by default |
| Native tools | Unsupported | Configure local fallbacks for web capabilities |
| Usage | Full feature support | Audio-token buckets and billable_audio_seconds in RunUsage.details |
| State-restoring reconnect | Full feature support | Native resumption is automatic with a reconnect policy |
See Audio, images, and transcripts, Turns and interruptions, Tools, and Connection lifecycle for the provider-agnostic workflows.
Grok Voice is not currently available through the Pydantic AI Gateway. Connect
through provider='xai' or an XaiProvider.
With a [ReconnectPolicy][pydantic_ai.realtime.ReconnectPolicy], xAI automatically enables native
resumption for state-restoring reconnects: it restores prior
turns and suppresses the provider's replay burst from the local event stream. The handle stays in
memory and cannot resume in another process.
supports_text_output=False, so output_modality='text'
raises a UserError before connecting. Read the answer from the transcript on the SpeechPart.interrupt()
without played_ms.