ADR-304 — Local Meta LLM Proxy Product

• Status: Proposed
• Date: 2026-07-10
• Deciders: ruflo core
• Related: ADR-302 (enrollment entry point), ADR-303 (exhaustion entry point), ADR-305 (funnel overview), ADR-306 (auth), ADR-307 (runtime, packaging, service lifecycle), ADR-308 (API contract), ADR-148 / ADR-149 (cost-optimal routing the proxy builds on), ADR-150 (optional-dependency + removability constraint this must satisfy)

This ADR defines the product: what the proxy does, its data-plane semantics, and its consent gates. The deployable runtime — binary, packaging, bind semantics, platform services, update integrity — is defined in ADR-307.

Context

Many RuFlo users already run local models or use multiple providers. Managing endpoints, API keys, and routing policies individually increases friction.

Cognitum provides Meta LLM orchestration through https://api.cognitum.one. A local proxy can expose a single OpenAI-compatible endpoint while transparently routing requests to the optimal provider — the same tier-routing discipline the repo already applies internally (3-tier model routing, metallm_ask/metallm_delegate gateway delegation, ADR-149 cost-optimal routing).

Decision

Offer an optional local proxy during onboarding (ADR-302), on credit exhaustion (ADR-303), and on demand via ruflo proxy install / ruflo proxy enable.

Architecture

Client (any OpenAI-compatible SDK / ruflo agents)
  ↓
localhost:11435
  ↓
Meta Proxy (local process, ruflo-managed)
  ↓
api.cognitum.one
  ↓
Claude │ GPT │ Gemini │ DeepSeek │ OpenRouter │ Local Ollama │ vLLM │ SGLang

Local backends (Ollama, vLLM, SGLang) are routed to directly by the local proxy without a cloud round-trip; api.cognitum.one is in the path only for cloud providers and for routing-policy updates.

Capabilities

• OpenAI-compatible API surface
• Automatic routing (difficulty-tiered, cheap-tier-first — same policy family as cognitum-auto)
• Cost optimization
• Latency optimization
• Retry policies
• Provider failover
• Request receipts (metered cost + resolved tier/model returned in-band, matching the metallm_ask contract)
• Local caching
• Future harness-evolution integration (ADR-150/151 surfaces)

Authentication

ruflo auth login

obtains credentials for proxy operation.

Data-plane disclosure (cloud routing is off by default)

"Local proxy" is easily read as "local inference." The two must never be conflated:

• Default state after ruflo proxy install is local-only. The proxy routes exclusively to local backends (Ollama, vLLM, SGLang). No prompt leaves the machine, and no request is made to api.cognitum.one for inference.

• Cloud routing requires a separate explicit step — ruflo proxy config --cloud — gated on the cloud-routing consent domain (ADR-302). Neither enrollment acceptance, auth login, nor proxy installation enables it.

• Pre-activation disclosure is mandatory. Before cloud routing turns on, the UI states in plain terms what changes:

  Copy

  Enabling cloud routing.
  
  With cloud routing ON, prompts for cloud-tier requests are sent to
  api.cognitum.one and forwarded to the selected provider
  (Claude / GPT / Gemini / DeepSeek / OpenRouter).
  
  Requests routed to local backends never leave this machine.
  
  Enable cloud routing? [y/N]
  

  The default answer is No.

• Visible at runtime. ruflo proxy status and every request receipt state the data plane used (local vs cloud:<provider>), so the user can verify where any given prompt went.

• Cloud routing can be disabled at any time (ruflo proxy config --local-only), reverting to a purely local multi-backend router and revoking the cloud-routing consent receipt.

Relationship to the metallm dev-bridge

The repository already carries an internal meta-llm gateway surface (metallm_ask / metallm_delegate, the dev-bridge MCP server). This proxy is related but not the same thing, and the boundary is explicit:

• The shared routing core is a library boundary; the dev-bridge and the proxy consume it independently.
• A compatibility layer, explicitly versioned, mediates anywhere the two must interoperate — the internal dev-bridge never becomes the de facto public contract, and public-contract changes never break internal tooling silently.
• Deprecating or changing the dev-bridge has no effect on proxy customers, and vice versa.

Constraints

• Optional and removable (ADR-150 discipline): the proxy ships as an optional component; ruflo remains fully operational with it absent or uninstalled. No dependencies entry — install is an explicit user action.
• No credentials in the repo or config files: tokens live in the OS keychain where available, else ~/.ruflo/credentials with 0600 permissions; never in project config, never committed (existing @claude-flow/security boundary rules apply).
• Local-first privacy posture: prompts routed to local backends never leave the machine; the cloud path is explicit and visible in request receipts.
• Default port 11435 (adjacent to Ollama's 11434, non-conflicting), configurable.
• Failure isolation: if the proxy is down, clients get a normal connection error — the proxy must never silently fall back from local-only mode to cloud routing.

Consequences

• New CLI surface: ruflo proxy … — full lifecycle command set (install|start|stop|status|logs|update|uninstall) specified in ADR-307, plus proxy config for routing mode.
• ruflo doctor gains a proxy health check component (details in ADR-307).
• This is the conversion product the ADR-301/302/303 touchpoints funnel toward; activation rate is a North Star metric in ADR-305.