Adaptive Router (v0)

A request-type-aware routing strategy. For each incoming request, classify the prompt into one of seven RequestType buckets (code generation, writing, analytical reasoning, …), then Thompson-sample a Beta(α, β) bandit posterior per (request_type, model) cell to pick the best model. Quality estimates are combined with a normalized cost score via a weighted linear sum.

A post-call hook reads the response and runs lightweight regex + tool-call detectors (see signals.py) to award per-turn credit/blame to the model that served the turn. Updates are batched in-memory and flushed to Postgres every ~10s by a background task in proxy_server.py.

Config example

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
    model_info:
      input_cost_per_token: 0.0000025
      adaptive_router_preferences:
        quality_tier: 3
        strengths: ["code_generation", "analytical_reasoning"]

  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
    model_info:
      input_cost_per_token: 0.00000015
      adaptive_router_preferences:
        quality_tier: 2
        strengths: ["general", "factual_lookup"]

  - model_name: smart-router
    litellm_params:
      model: auto_router/adaptive_router
      adaptive_router_default_model: gpt-4o-mini
      adaptive_router_config:
        available_models: ["gpt-4o", "gpt-4o-mini"]
        weights:
          quality: 0.7
          cost: 0.3

Callers may pass header x-litellm-min-quality-tier: 3 (or metadata key min_quality_tier: 3) to force selection from tier-3-or-higher models only.

Behavior summary

• Cold start. Each (request_type, model) cell starts with a Beta prior whose mean = BASE_TIER_WEIGHT[tier] (+ STRENGTH_BONUS if declared) and total mass = COLD_START_MASS (10). About ten real observations move it meaningfully.
• Per-request decision. Sample once per eligible model, score with quality_weight·sample + cost_weight·normalized_cost, pick the argmax. Routing is stateless per-turn — no sticky lookup. Each call resamples.
• Previous-response attribution. Post-call, feedback from the current user message is attributed to the model that produced the previous response, while response signals are attributed to the current model. Contexts expire after 24 hours and the in-memory cache is capped at 1,024 sessions. Conversation identity is the client-supplied litellm_session_id if present, otherwise a sha256 over caller identity (api key hash, team, user, end-user) + the first message.
• Per-turn updates. satisfaction → +α. misalignment, stagnation, disengagement, failure → +β (each). loop → +0.5β. exhaustion → 0 (uptime, not quality). Skipped if conversation has fewer than SIGNAL_GATE_MIN_MESSAGES messages.
• Persistence. Bandit cells: aggregated deltas, eventually consistent. Session rows: last-write-wins snapshots.

Known v0 limitations

• Latency is not in the score. Quality + cost only. A pathologically slow model can still be picked.
• Hard sample cap at 200. Once α + β > 200, deltas are silently dropped. No rescaling — drift is a v1 concern.
• Signals are regex + tool-call only. No LLM-judge, no embedding similarity, no exemplar storage. Signals are best-effort and biased toward English.
• One AdaptiveRouter per Router. Multiple adaptive_router/* deployments on the same litellm.Router raise at init.
• Bandit-delta mapping is unvalidated. _compute_bandit_delta is a v0 guess; expect to retune after the first ~1000 sessions of real traffic.
• request_type is classified per turn from the latest user message. For non-GENERAL turns, the current-turn type is used for bandit attribution (so genuine mid-session topic shifts update the correct cell). For GENERAL turns ("thanks!", "ok", "sounds good"), attribution falls back to the session's original type to avoid misattributing closing pleasantries.