docs/concepts/model-failover.md
OpenClaw handles failures in two stages:
agents.defaults.model.fallbacks.This is intentionally narrower than "save and restore the whole session." The reply runner only persists the model-selection fields it owns for fallback: providerOverride, modelOverride, modelOverrideSource, authProfileOverride, authProfileOverrideSource, authProfileOverrideCompactionCount. That prevents a failed fallback retry from overwriting newer unrelated session mutations, such as a manual /model change or a session rotation update that happened while the attempt was running.
The selection source controls whether the fallback chain is allowed:
agents.defaults.model.primary uses agents.defaults.model.fallbacks.agents.list[].model is strict unless that agent's model object includes its own fallbacks. Use fallbacks: [] to make the strict behavior explicit, or a non-empty list to opt that agent into model fallback.providerOverride, modelOverride, modelOverrideSource: "auto", and the selected origin model before retrying. This override keeps walking the configured fallback chain without probing the primary on every message, but OpenClaw probes the configured origin every 5 minutes (not configurable) and clears the override once it recovers. /new, /reset, and sessions.reset also clear auto-sourced overrides. Heartbeat runs without an explicit heartbeat.model clear direct auto overrides when their origin no longer matches the current configured default./model, the model picker, session_status(model=...), and sessions.patch write modelOverrideSource: "user". This is an exact session selection. If the selected provider/model fails before producing a reply, OpenClaw reports the failure instead of answering from an unrelated configured fallback.modelOverride without modelOverrideSource. OpenClaw treats those as user overrides so an explicit old selection is not silently converted into fallback behavior.payload.model / --model is a job primary, not a user session override. It uses configured fallbacks unless the job provides payload.fallbacks; payload.fallbacks: [] makes the cron run strict.OpenClaw remembers recent primary probes per session and primary model so a failing primary is not retried on every turn. It sends a visible notice when a session moves onto fallback and another notice when it returns to the selected primary; it does not repeat the notice on every sticky fallback turn.
By default, every new turn keeps the existing fallback retry behavior: OpenClaw retries each configured fallback candidate again, including non-primary candidates that recently failed with auth or auth_permanent.
Opt in to suppress repeat auth failures with:
OPENCLAW_FALLBACK_SKIP_TTL_MS=60000
When enabled, OpenClaw records an in-memory, session-scoped skip marker for a non-primary fallback candidate after an auth-class failure, keyed by session id, provider, and model. Primary candidates are never skipped, so an explicit user model selection still surfaces the real auth error. The cache is process-local and clears on Gateway restart.
The value is a TTL in milliseconds. 0 or unset disables the cache. Positive values are clamped between 1 second and 10 minutes.
When a session moves onto an auto-selected fallback, OpenClaw sends a status notice in the same reply surface:
↪️ Model Fallback: <fallback> (selected <primary>; <reason>)
When a later probe succeeds and the session returns to the selected primary, OpenClaw sends:
↪️ Model Fallback cleared: <primary> (was <fallback>)
These notices are operational messages, not assistant content. They deliver once per state change, including side-effect-only turns when feasible, but sticky fallback turns do not repeat them. Delivery bypasses normal source-reply suppression, does not consume the first assistant reply slot for threaded channels, and is excluded from text-to-speech and commitment extraction.
OpenClaw uses auth profiles for both API keys and OAuth tokens.
~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite.auth.profiles / auth.order are metadata + routing only (no secrets).~/.openclaw/credentials/oauth.json (imported into the per-agent auth store on first use).auth-profiles.json, auth-state.json, and per-agent auth.json files are imported by openclaw doctor --fix.More detail: OAuth
Credential types:
type: "api_key" → { provider, key }type: "oauth" → { provider, access, refresh, expires, email? } (+ projectId/enterpriseUrl for some providers)type: "token" → static bearer-style token, optionally expiring; OpenClaw does not refresh it (used for aws-sdk and other credential-chain auth modes)OAuth logins create distinct profiles so multiple accounts can coexist.
provider:default when no email is available.provider:<email> (for example google-antigravity:[email protected]).Profiles live in the per-agent openclaw-agent.sqlite auth profile store.
When a provider has multiple profiles, OpenClaw chooses an order like this:
<Steps> <Step title="Explicit config"> `auth.order[provider]` (if set). </Step> <Step title="Configured profiles"> `auth.profiles` filtered by provider. </Step> <Step title="Stored profiles"> Per-agent SQLite auth profile entries for the provider. </Step> </Steps>If no explicit order is configured, OpenClaw uses a round-robin order:
usageStats.lastUsed (oldest first, within each type).OpenClaw pins the chosen auth profile per session to keep provider caches warm. It does not rotate on every request. The pinned profile is reused until:
/new / /reset)Manual selection via /model …@<profileId> sets a user override for that session and is not auto-rotated until a new session starts.
For OpenAI agent models, auth and runtime are separate. openai/gpt-* stays on the Codex harness while auth can rotate between a Codex subscription profile and an OpenAI API-key backup.
Use auth.order.openai for the user-facing order:
{
auth: {
order: {
openai: ["openai:[email protected]", "openai:api-key-backup"],
},
},
}
Use openai:* for both ChatGPT/Codex OAuth profiles and OpenAI API-key profiles. When the subscription hits a Codex usage limit, OpenClaw records the exact reset time when Codex provides one, tries the next ordered auth profile, and keeps the run inside the Codex harness. Once the reset time passes, the subscription profile is eligible again and the next automatic selection can return to it.
Use a user-pinned profile only when you want to force one account/key for that session. User-pinned profiles are intentionally strict and do not silently jump to another profile.
When a profile fails due to auth/rate-limit errors (or a timeout that looks like rate limiting), OpenClaw marks it in cooldown and moves to the next profile.
<AccordionGroup> <Accordion title="What lands in the rate-limit / timeout bucket"> That rate-limit bucket is broader than plain `429`: it also includes provider messages such as `Too many concurrent requests`, `ThrottlingException`, `concurrency limit reached`, `workers_ai ... quota limit exceeded`, `throttled`, `resource exhausted`, and periodic usage-window limits such as `weekly limit reached` or `monthly limit exhausted`.Format/invalid-request errors are usually terminal because retrying the same payload would fail the same way, so OpenClaw surfaces them instead of rotating auth profiles. Known retry-repair paths can opt in explicitly: for example Cloud Code Assist tool call ID validation failures are sanitized and retried once through the `allowFormatRetry` policy. OpenAI-compatible stop-reason errors such as `Unhandled stop reason: error`, `stop reason: error`, and `reason: error` are classified as timeout/failover signals.
Generic server text can also land in that timeout bucket when the source matches a known transient pattern. For example, the bare model runtime stream-wrapper message `An unknown error occurred` is treated as failover-worthy for every provider because the shared model runtime emits it when provider streams end with `stopReason: "aborted"` or `stopReason: "error"` without specific details. JSON `api_error` payloads with transient server text such as `internal server error`, `unknown error, 520`, `upstream error`, or `backend error` are also treated as failover-worthy timeouts.
OpenRouter-specific generic upstream text such as bare `Provider returned error` is treated as timeout only when the provider context is actually OpenRouter. Generic internal fallback text such as `LLM request failed with an unknown error.` stays conservative and does not trigger failover by itself.
- OpenClaw records `cooldownModel` for rate-limit failures when the failing model id is known.
- A sibling model on the same provider can still be tried when the cooldown is scoped to a different model.
- Billing/disabled windows still block the whole profile across models.
Regular (non-billing, non-auth-permanent) cooldowns scale with the profile's recent error count:
Counters reset once the profile's failure window has passed (auth.cooldowns.failureWindowHours, default 24).
State is stored in the per-agent SQLite auth state under usageStats:
{
"usageStats": {
"provider:profile": {
"lastUsed": 1736160000000,
"cooldownUntil": 1736160600000,
"errorCount": 2
}
}
}
Billing/credit failures (for example "insufficient credits" / "credit balance too low") are treated as failover-worthy, but they're usually not transient. Instead of a short cooldown, OpenClaw marks the profile as disabled (with a longer backoff) and rotates to the next profile/provider.
<Note> Not every billing-shaped response is `402`, and not every HTTP `402` lands here. OpenClaw keeps explicit billing text in the billing lane even when a provider returns `401` or `403` instead, but provider-specific matchers stay scoped to the provider that owns them (for example OpenRouter `403 Key limit exceeded`).Meanwhile temporary 402 usage-window and organization/workspace spend-limit errors are classified as rate_limit when the message looks retryable (for example weekly usage limit exhausted, daily limit reached, resets tomorrow, or organization spending limit exceeded). Those stay on the short cooldown/failover path instead of the long billing-disable path.
</Note>
High-confidence permanent-auth failures (revoked/deactivated keys, deactivated workspaces) get a similar disabled lane, but recover much sooner than billing since some providers surface auth-looking payloads transiently during incidents.
State is stored in the per-agent SQLite auth state:
{
"usageStats": {
"provider:profile": {
"disabledUntil": 1736178000000,
"disabledReason": "billing"
}
}
}
Defaults (auth.cooldowns.*):
| Key | Default | Purpose |
|---|---|---|
billingBackoffHours | 5 | Base billing backoff, doubles per billing failure |
billingMaxHours | 24 | Billing backoff cap |
authPermanentBackoffMinutes | 10 | Base backoff for high-confidence permanent-auth failures |
authPermanentMaxMinutes | 60 | Cap for that backoff |
failureWindowHours | 24 | Failure counters reset if no failures occur in this window |
overloadedProfileRotations | 1 | Same-provider profile rotations allowed before model fallback on overload |
overloadedBackoffMs | 0 | Fixed delay before an overloaded rotation retry |
rateLimitedProfileRotations | 1 | Same-provider profile rotations allowed before model fallback on rate limit |
Overloaded and rate-limit errors are handled more aggressively than billing cooldowns: by default, OpenClaw allows one same-provider auth-profile retry, then switches to the next configured model fallback without waiting.
If all profiles for a provider fail, OpenClaw moves to the next model in agents.defaults.model.fallbacks. This applies to auth failures, rate limits, and timeouts that exhausted profile rotation (other errors do not advance fallback). Provider errors that do not expose enough detail are still labeled precisely in fallback state: empty_response means the provider returned no usable message or status, no_error_details means the provider explicitly returned Unknown error (no error details in response), and unclassified means OpenClaw preserved the raw preview but no classifier matched it yet.
Provider-busy signals such as ModelNotReadyException land in the overloaded bucket and follow the same one-rotation-then-fallback policy as rate limits (see the defaults table above).
When a run starts from the configured default primary, a cron job primary, an agent primary with explicit fallbacks, or an auto-selected fallback override, OpenClaw can walk the matching configured fallback chain. Agent primaries without explicit fallbacks and explicit user selections (for example /model ollama/qwen3.5:27b, the model picker, sessions.patch, or one-off CLI provider/model overrides) are strict: if that provider/model is unreachable or fails before producing a reply, OpenClaw reports the failure instead of answering from an unrelated fallback.
OpenClaw builds the candidate list from the currently requested provider/model plus configured fallbacks.
When every auth profile for a provider is already in cooldown, OpenClaw does not automatically skip that provider forever. It makes a per-candidate decision:
<AccordionGroup> <Accordion title="Per-candidate decisions"> - Persistent auth failures skip the whole provider immediately. - Billing disables usually skip, but the primary candidate can still be probed on a throttle so recovery is possible without restarting. - The primary candidate may be probed near cooldown expiry, with a per-provider throttle. - Same-provider fallback siblings can be attempted despite cooldown when the failure looks transient (`rate_limit`, `overloaded`, or unknown). This is especially relevant when a rate limit is model-scoped and a sibling model may still recover immediately. - Transient cooldown probes are limited to one per provider per fallback run so a single provider does not stall cross-provider fallback. </Accordion> </AccordionGroup>Session model changes are shared state. The active runner, /model command, compaction/session updates, and live-session reconciliation all read or write parts of the same session entry.
That means fallback retries have to coordinate with live model switching:
/model, session_status(model=...), and sessions.patch.agents.defaults.model.fallbacks./new, /reset, and sessions.reset clear auto-sourced overrides immediately./status shows the selected model and, when fallback state differs, the active fallback model and reason.This prevents the classic race:
<Steps> <Step title="Primary fails"> The selected primary model fails. </Step> <Step title="Fallback chosen in memory"> Fallback candidate is chosen in memory. </Step> <Step title="Session store still says old primary"> Session store still reflects the old primary. </Step> <Step title="Live reconciliation reads stale state"> Live-session reconciliation reads the stale session state. </Step> <Step title="Retry snapped back"> The retry gets snapped back to the old model before the fallback attempt starts. </Step> </Steps>The persisted fallback override closes that window, and the narrow rollback keeps newer manual or runtime session changes intact.
runWithModelFallback(...) records per-attempt details that feed logs and user-facing cooldown messaging:
rate_limit, overloaded, billing, auth, model_not_found, and similar failover reasons)Structured model_fallback_decision logs also include flat fallbackStep* fields when a candidate fails, is skipped, or a later fallback succeeds. These fields make the attempted transition explicit (fallbackStepFromModel, fallbackStepToModel, fallbackStepFromFailureReason, fallbackStepFromFailureDetail, fallbackStepFinalOutcome) so log and diagnostic exporters can reconstruct the primary failure even when the terminal fallback also fails.
When every candidate fails, OpenClaw throws FallbackSummaryError. The outer reply runner can use that to build a more specific message such as "all models are temporarily rate-limited" and include the soonest cooldown expiry when one is known.
That cooldown summary is model-aware:
See Gateway configuration for:
auth.profiles / auth.orderauth.cooldowns.billingBackoffHours / auth.cooldowns.billingBackoffHoursByProviderauth.cooldowns.billingMaxHours / auth.cooldowns.failureWindowHoursauth.cooldowns.authPermanentBackoffMinutes / auth.cooldowns.authPermanentMaxMinutesauth.cooldowns.overloadedProfileRotations / auth.cooldowns.overloadedBackoffMsauth.cooldowns.rateLimitedProfileRotationsagents.defaults.model.primary / agents.defaults.model.fallbacksagents.defaults.imageModel routingSee Models for the broader model selection and fallback overview.