Back to Omniroute

Admission lanes — two lane systems, what gates each, where each reports

docs/architecture/admission-lanes.md

3.8.492.5 KB
Original Source

Admission lanes (#9654) — two lane systems, what gates each, where each reports

OmniRoute has two process-local lane systems with different scopes. They are complementary; operators should know which one they are looking at.

1. Byte-level per-connection lanes (chatBodyAdmission.ts)

  • Scope: the buffered-body/heap path for POST /v1/chat/completions. Guards against heap amplification from large coding-agent bodies (#4380).
  • Gate: always on. Each distinct API key (hashed) — or anonymous — gets its own lane with CHAT_MAX_HEAVY_IN_FLIGHT capacity, so one session's burst cannot starve another session's heavyweight slot.
  • Tuning:
    • OMNIROUTE_CHAT_VIRTUAL_TTL_MS — idle-lane eviction (default 60000)
    • OMNIROUTE_CHAT_VIRTUAL_MAX_SESSIONS — lane count cap (default 64)
    • OMNIROUTE_CHAT_ADMISSION_QUEUE_MS — queue-wait before 503 (default 2000)
    • OMNIROUTE_CHAT_ADMISSION_MAX_QUEUED_BYTES — queued-bytes heap valve (default 4 MB)
  • Reports: not in GET /api/monitoring/health today; observable via PerConnectionAdmissionController.snapshot() (sessionId hash, activeHeavy, idleMs).

2. Adaptive runtime virtual lanes (open-sse/services/admission)

  • Scope: tenant-key admission for provider dispatch — queue cost, latency-guided limit adaptation, lane queueing, and lane metrics.
  • Gate: opt-in. Disabled unless OMNIROUTE_CHAT_VIRTUAL_LANES=true. Without it, the adaptive controller keeps the shared queue behavior (criterion 1 of #9654 only holds once an operator enables lanes).
  • Tuning: OMNIROUTE_CHAT_VIRTUAL_LANES + adaptive config (maxQueueCount, maxQueueCost, defaultMaxWaitMs, …).
  • Reports: GET /api/monitoring/healthadaptiveAdmissionlaneCount, laneQueuedCount, laneQueuedCost, laneTenants (opaque lane IDs, never raw keys).

Which one is showing in a dashboard

  • adaptiveAdmission.laneCount / laneTenantsadaptive virtual lanes (system 2).
  • A health payload with no adaptiveAdmission.lane* fields usually means OMNIROUTE_CHAT_VIRTUAL_LANES is unset — the byte-level lanes (system 1) are still active, but nothing under adaptiveAdmission will report lane data until it is enabled.

Why both exist

The byte-level lanes bound the memory-heavy parse/compress path; the adaptive lanes bound dispatch cost per tenant. #9654's criterion 1 ("one session's burst does not 503 another") is enforced by system 1 unconditionally and by system 2 once opt-in is enabled.