docs/architecture/RESILIENCE_GUIDE.md
OmniRoute has three distinct but related resilience mechanisms. Each has a different scope and purpose. Keep them separate when debugging routing behavior.
Source: diagrams/resilience-3layers.mmd
Scope: entire provider (e.g., glm, openai, anthropic).
Purpose: stop sending traffic to a provider that is repeatedly failing at the upstream/service level.
Implementation:
src/shared/utils/circuitBreaker.tssrc/sse/handlers/chatHelpers.ts, src/sse/handlers/chat.tsGET /api/monitoring/healthPOST /api/resilience/resetopen-sse/services/accountFallback.tsdomain_circuit_breakersStates:
CLOSED — normal traffic allowedDEGRADED — traffic still allowed, but elevated provider failures are being trackedOPEN — provider temporarily blocked; combo routing skips itHALF_OPEN — reset timeout elapsed; probe request allowedConfigurable defaults (open-sse/config/constants.ts, exposed in Dashboard → Settings → Resilience):
| Class | Degraded at | Opens at | Reset timeout |
|---|---|---|---|
| OAuth | 5 failures | 8 failures | 60s |
| API-key | 7 failures | 12 failures | 30s |
| Local | derived | 2 failures | 15s |
degradationThreshold controls when a provider enters DEGRADED; failureThreshold controls when it opens and is skipped. Local provider profiles are not exposed on the Resilience settings page yet.
Trip codes: only provider-level statuses [408, 500, 502, 503, 504]. Do NOT trip for account-level errors (most 401/403/429 — those belong to cooldown or lockout).
Lazy recovery: when OPEN expires, getStatus(), canExecute(), getRetryAfterMs() refresh state to HALF_OPEN. No background timer needed.
Scope: single provider connection/account/key.
Purpose: skip one bad key while other connections for the same provider keep serving.
Implementation:
src/sse/services/auth.ts::markAccountUnavailable()getProviderCredentials* in same fileopen-sse/services/accountFallback.ts::checkFallbackError()src/lib/resilience/settings.tsFields per connection:
rateLimitedUntil — timestamp until cooldown expirestestStatus: "unavailable"lastError, lastErrorType, errorCodebackoffLevel — exponential backoff counterDefault cooldowns:
Retry-After/reset headers/parseable reset textbaseCooldownMs * 2 ** failureIndexAnti-thundering-herd guard: prevents concurrent failures from over-extending cooldown or double-incrementing backoffLevel.
Terminal states (NOT cooldowns):
banned — set by banned-keyword / account-ban detection (see BAN_DETECTION)expiredcredits_exhaustedThese persist until credentials change or an operator resets them. Do not overwrite terminal states with transient cooldown state.
Lazy recovery: when rateLimitedUntil is past, connection becomes eligible again. On successful use, clearAccountError() clears all error fields.
Scope: one client session (X-Session-Id / x-codex-session-id / x-omniroute-session header) pinned to one connection, for any provider.
Purpose: keep a multi-turn agent (Claude Code, aider, custom agents) on the same account across requests, reducing cross-account context loss and repeated cold-start 429s on providers with per-account session state.
Implementation:
src/sse/services/sessionAffinityPin.ts::resolveSessionAffinityTtlMs()src/sse/services/sessionAffinityPin.ts::selectSessionAffinityConnection()src/sse/services/auth.ts::extractSessionAffinityKey()sessionAccountAffinity (src/lib/db/sessionAccountAffinity.ts)sessionAffinityTtlMs (global TTL in ms, 0 disables) — src/lib/db/settings.ts. Renamed from the Codex-only codexSessionAffinityTtlMs by migration 124_generic_session_affinity_ttl.sql, which carries over any previously-configured Codex TTL as the new default.Before #7274, resolveSessionAffinityTtlMs() hard-bailed to 0 for every provider except codex, so the TTL setting (and the session headers) had no effect anywhere else even though the pinning mechanism and header extraction were already provider-agnostic. The fix removed that early-return; the TTL now applies uniformly to every provider once set globally above 0.
The three session-affinity headers are never forwarded upstream — executors build their own upstream headers from scratch rather than passing client headers through, so this stays an internal correlation id only.
Scope: provider + connection + model triple.
Purpose: avoid disabling a whole connection when only one model is unavailable or quota-limited.
Examples:
Implementation: open-sse/services/accountFallback.ts — lockModel(), clearModelLock(), getAllModelLockouts().
UI: Settings → Model Cooldowns (src/app/(dashboard)/dashboard/settings/components/ModelCooldownsCard.tsx)
Lists active lockouts with: provider, connection, model, reason, expiresAt. Operators can manually re-enable a model from the card.
REST API:
GET /api/resilience/model-cooldowns — list active lockoutsDELETE /api/resilience/model-cooldowns — manual re-enable. Body: {provider, connection, model}. Auth: management.Model lockout went from always-on hardcoded behavior to a fully configurable, opt-in feature with its own settings card and a self-healing recovery path.
Settings card: Settings → Model Lockout
(src/app/(dashboard)/dashboard/settings/components/ModelLockoutCard.tsx).
This is distinct from the read-only ModelCooldownsCard above (which only
lists active lockouts) — the new card configures the parameters. Defaults
live in DEFAULT_MODEL_LOCKOUT_SETTINGS
(src/lib/resilience/modelLockoutSettings.ts):
| Setting | Default | Meaning |
|---|---|---|
enabled | false | Master toggle — model lockout is off by default. |
errorCodes | [403, 404, 429, 502, 503, 504] | Upstream statuses that count as a model-scoped failure. |
baseCooldownMs | 120_000 (120 s) | Initial lockout duration for the first failure. |
maxCooldownMs | 1_800_000 (30 min) | Cap on the escalated cooldown. |
maxBackoffSteps | 10 | Max exponential-backoff escalation steps. |
useExponentialBackoff | true | Whether repeated failures escalate the cooldown exponentially. |
Settings persist through the normal settings store and validate via the
resilience settings schema; the card clamps baseCooldownMs/maxCooldownMs
(with maxCooldownMs ≥ baseCooldownMs) and maxBackoffSteps.
Success-decay recovery: recovery is not purely timer expiry. A healthy
response walks the model's failure count back down so a model that recovered
mid-window stops escalating (and clears) before its timer would. On a successful
combo target, open-sse/services/combo.ts calls decayModelFailureCount()
(open-sse/services/accountFallback.ts), which halves the stored
failureCount (Math.floor(failureCount / 2)); when it reaches 0 the lockout
entry is deleted entirely. The counterpart recordModelLockoutFailure()
increments the count (and escalates the cooldown) on failures within the
escalation window. This success-decay is in addition to plain timer expiry —
either path can re-enable a model.
State: lockouts are held in-memory (per-process Maps of
ModelLockoutEntry keyed by provider:connectionId:model), not persisted to
the DB — they are lost on restart. The settings are persisted; the active
lockout state is ephemeral.
Subscription accounts (GLM, MiniMax, etc.) often accept only ~1–3 concurrent
requests; exceeding that triggers 429s and cooldowns. This is acute under
quota-share (qtSd/…) combos, where several API keys share one upstream
account. Three layers keep a shared account from being flooded.
max_concurrent)Each provider connection can declare a max_concurrent ceiling
(provider_connections.max_concurrent, set in the connection modal / API / DB).
Leave it empty for no limit. This is the single knob that drives the serialization
layer below — set it to the account's real concurrency (e.g. GLM ~1, MiniMax ~2).
When a quota-share dispatch targets a connection that declares a positive
max_concurrent, concurrent requests to that account are serialized through a
per-connection semaphore (key qsconn:<connectionId>): excess requests wait in
the queue instead of flooding the account. It is fail-open — a saturated
queue or timeout proceeds without a slot rather than ever rejecting a dispatchable
request. Toggle in Settings → Resilience → Quota-share per-connection
concurrency (resilienceSettings.quotaShareConcurrencyLimit.enabled, default
on). Without a max_concurrent cap the behavior is unchanged.
The quota-share routing gate (
selectQuotaShareTarget, DRR + P2C) is itself fail-open and only deprioritizes an at-cap connection — with a single-connection pool it cannot hard-limit, so this semaphore is what actually contains the flood.
For every combo strategy (when enabled), a request that would crystallize a 429
for a SHORT transient cooldown waits it out and re-dispatches instead of
returning the 429 — this covers Gemini-class TPM/RPM windows (~60s retry-after)
on multi-model combos, e.g. both targets of a 2-model combo hitting a per-model
rate limit. Bounded by comboCooldownWait (enabled, maxWaitMs, maxAttempts,
budgetMs) in Settings → Resilience. It never waits on quota_exhausted
(locked until midnight) or auth/not-found reasons.
Scope: the local per-provider+connection rate-limit queue (open-sse/services/rateLimitManager.ts,
backed by Bottleneck), one layer below the three mechanisms above.
maxWaitMs is a legacy persisted name for execution expiration.
resilienceSettings.requestQueue.maxWaitMs is passed to Bottleneck as a job
expiration, whose timer starts only after dispatch. It therefore bounds
limiter-managed execution, not time spent in the local queue. Expiration is
surfaced as trusted local code: "RATE_LIMIT_EXECUTION_TIMEOUT" (HTTP 504);
the former queue-timeout code name is accepted only for trusted internal
backward compatibility. The default is 15000ms; override via
RATE_LIMIT_MAX_WAIT_MS (env) or the dashboard (Settings → Resilience,
1–30000ms UI ceiling). Queue residence has no time deadline; use
maxQueueDepth below to bound queued callers.
maxQueueDepth — opt-in admission cap (new). resilienceSettings.requestQueue.maxQueueDepth
bounds how many requests may sit queued (not yet dispatched) for one
provider+connection at once. When the queue already holds maxQueueDepth
requests, a new request is fast-rejected with a typed
code: "RATE_LIMIT_QUEUE_FULL" error before it ever reaches limiter.schedule()
— so the rejection is cheap and happens ahead of any downstream
prompt-compression / translation work for that request. Default 0 =
disabled, preserving the existing unbounded-queue behavior; bounded 0–100000.
Override via RATE_LIMIT_MAX_QUEUE_DEPTH (env) or
resilienceSettings.requestQueue.maxQueueDepth (dashboard/API patch).
The admission check itself is a pure function
(open-sse/services/rateLimitManager/admission.ts::checkQueueAdmission) so
it is unit-testable without a real Bottleneck limiter.
The RFC that opened #6593 also proposed a
bypassCompressionOnRateLimitflag. This repo'sopen-sse/services/compression/pipeline is prompt/context compression on the outbound LLM request (chatCore.ts, around theresolveCompressionSettings/selectCompressionStrategyblock), not HTTP response compression on synthesized 429 bodies — there is no matching code path for a literal bypass flag. That prompt-compression step also currently runs beforewithRateLimit()in the request pipeline, so reordering to skip it on a queue-full rejection is a separate, larger change than this issue's scope; it was intentionally not implemented here and is left as a follow-up if the CPU-saving win is worth the reordering risk.
The optional resilienceSettings.streamRecovery.throughputWatchdog guard detects
an upstream that is still sending chunks but producing assistant output below the
configured useful-output rate. It is deliberately distinct from the idle timeout:
heartbeats and metadata reset neither timer and do not count as progress. It is also
distinct from the hard attempt deadline (#9153), which remains an absolute safety
ceiling regardless of output quality.
The watchdog requires a warm-up period followed by a complete rolling window before
it can abort. It counts text deltas from Chat Completions and Responses API output
events (a conservative UTF-8 byte proxy), ignores usage-only and empty events, and
suspends judgement while tool-call or reasoning events are in flight. It is disabled
by default and can be enabled with STREAM_THROUGHPUT_WATCHDOG_ENABLED=true; the
window, warm-up, minimum rate, and minimum measurable output are bounded by the
normal resilience-settings normalization layer.
When enabled, a watchdog abort is applied only to the active upstream attempt. Before any client-visible bytes, the existing same-account early-recovery path may reopen the attempt. After commit, the stream is never blindly replayed; only the existing safe mid-stream continuation contract can stitch a suffix. Finalization remains single-shot, so usage accounting and semaphore release are not duplicated.
Scope: one upstream gateway that reports temporary quota exhaustion with the wrong HTTP status.
Purpose: correct a misleading status BEFORE classification, so downstream consumers (fallback engine, combo aggregation, the client-facing response) see the true retryable nature of the failure.
Some gateways signal TEMPORARY quota exhaustion with a non-retryable HTTP
status. agentrouter.org returns 403 (sometimes 400) with a Chinese body
(用户额度不足 / 额度不足) instead of the standard 429. Clients like Claude
Code treat 403 as permanent and abort the session, and without correction
the fallback engine would classify it as AUTH_ERROR instead of a quota
event.
Implementation:
open-sse/config/upstreamStatusRestatement.ts — a
per-provider list of rules ({id, fromStatuses, toStatus, textMarkers, excludeMarkers, defaultRetryAfterMs}), matched via applyStatusRestatement().providerFailure: block in open-sse/handlers/chatCore.ts
(around line 3654), right after parseUpstreamError() parses an upstream
response with an error HTTP status (!providerResponse.ok), and before any
classification runs, so every downstream consumer sees the corrected
status. Errors embedded inside a 200 SSE stream follow a separate,
later stream-parsing path and are not covered by this hook today — a
known limitation, not yet needed for agentrouter's misstatus (which
surfaces as an error HTTP status).429 is in RETRY_AFTER_ELIGIBLE_STATUSES
(open-sse/services/combo/unavailableRetryGate.ts), so a restated error
carries a real retry window instead of surfacing as a dead 403.60s defaultRetryAfterMs (upstreamStatusRestatement.ts)
is only what the restated response tells the client; it is not itself
the connection's internal cooldown/lockout duration — that is governed
separately by whichever mechanism actually handles the restated error
(Connection Cooldown's escalating backoff, §2, base 3s for API-key
providers; or Model Lockout, §3, for per-model-quota providers like
agentrouter). The router can become eligible to retry internally sooner
than the 60s window it advertises to the client — intentional headroom,
not a bug.Permanent errors (agentrouter's 无权访问模型 — no access to this model) are
NEVER restated: excludeMarkers vetoes the rule even when textMarkers hit,
so the error keeps its original status and nothing retries it forever. A
separate provider classification rule
(agentrouter-model-access-denied in open-sse/config/providerErrorRules.ts)
declares an auth_error/scope-model match for this text, but it does not
fire on the live production path today: the rule only matches status === 403, and checkFallbackError's apikey-category FORBIDDEN branch
(open-sse/services/accountFallback.ts) returns early for a plain 403
before the provider-rule lookup ever runs. In practice a 无权访问模型 403
is handled the same way as the base apikey-provider 403 path (see Connection
Cooldown, §2), not as a 6h model lockout. The rule still exists as a
declarative classification consumable by future callers of classifyError
with context — wiring it into the production checkFallbackError path is
tracked as a follow-up, not yet done.
Restated quota errors (额度不足) do reach a provider rule in production
(agentrouter-user-quota-exhausted, scope "connection"), but scope on
ProviderErrorRuleMatch is currently informational — the persistence path
(checkFallbackError → combo.ts) only consumes reason and cooldownMs,
never scope. What actually happens for agentrouter (passthroughModels: true → hasPerModelQuota() returns true) is a per-model lockout via
recordModelLockoutFailure(): the connection itself is never cooled down for
this error (combo.ts skips recordProviderCooldown for 429 when
hasPerModelQuota is true), so other models on the same account keep being
tried — each one burns one call and its own lockout before combo routing
moves on. Honoring scope end-to-end (so a "connection" match actually
locks the connection) is tracked as a follow-up.
Status restatement (upstreamStatusRestatement.ts) and provider
classification rules (open-sse/config/providerErrorRules.ts,
providerRuleRegistry) are separate registries that both key on provider id
and text markers, but they run in different places and serve different
purposes: restatement rewrites the HTTP status early in chatCore.ts;
classification rules pick the fallback reason and lock scope
(model / provider / connection) inside checkFallbackError()
(open-sse/services/accountFallback.ts).
Classification rules only see full error text (needed to match body
markers like 额度不足) for providers listed in the FULL_TEXT_RULE_PROVIDERS
allowlist in providerErrorRules.ts — currently only "agentrouter". For
every other provider, checkFallbackError hands getProviderErrorRuleMatch
only the structured error ({code, type}), which is enough for
header/status/code-based rules but blind to body-text markers. The helper
resolveRuleMatchBody() performs this selection: full error text for
allowlisted providers, the structured error otherwise. Adding a provider to
FULL_TEXT_RULE_PROVIDERS is an explicit per-provider opt-in — it exists so
that the default path for every provider not on the list stays
byte-for-byte unchanged.
statusRestatementRegistry
(open-sse/config/upstreamStatusRestatement.ts). Keep textMarkers
provider-specific; never reuse generic English phrases that collide with
CREDITS_EXHAUSTED_SIGNALS (open-sse/services/accountFallback.ts).open-sse/config/providerErrorRules.ts (providerRuleRegistry) to pick
the right lock scope (connection for account-wide quota, model for
per-model errors). This step only takes effect in production for
providers whose rules need the full error text (body markers): add the
provider id to FULL_TEXT_RULE_PROVIDERS in the same file — otherwise
checkFallbackError only ever hands the rule the structured
{code, type} error and a body-text rule will never match live traffic.
Rules that match purely on status/headers (like Opencode's or
Minimax's) do not need this opt-in.tests/unit/upstream-status-restatement.test.ts
and tests/unit/agentrouter-error-rules.test.ts (including the
not-permanent / not-creditsExhausted guards, and — if the provider needs
the allowlist — a test asserting resolveRuleMatchBody() returns the
full text only for that provider).No changes to chatCore.ts, classifyError, or combo are needed.
background: true degraded to sync with warning.OMNIROUTE_EMERGENCY_FALLBACK; operators can override it from the Feature Flags page without a restart.rateLimitedUntil/testStatus.state instead of getStatus()/canExecute().Provider-specific stealth (JA3/JA4, CCH, obfuscation) is separately documented — see STEALTH_GUIDE.md.
Beyond unit tests for resilience logic, three tests exercise the runtime under real stress/failure conditions (all integration/nightly — none block PRs):
| Test | What | Run |
|---|---|---|
| Chaos | Fake-upstream node injects real latency/reset/timeout/503; validates that the circuit breaker opens/recovers and checkFallbackError classifies 503 as recoverable fallback. | RUN_CHAOS_INT=1 npm run test:chaos |
| Heap-growth | ~500 streams per createSSEStream under --expose-gc; fails if the heap grows beyond the ceiling (OOM guard #3069). | npm run test:heap |
| k6 soak | Sustained load against /api/monitoring/health; p95/error thresholds. | k6 run tests/load/k6-soak.js (nightly) |
Orchestrated by .github/workflows/nightly-resilience.yml (cron + dispatch). In the
default test:integration, chaos and heap self-skip (without RUN_CHAOS_INT/--expose-gc).