Compression Engines

OmniRoute compression is built around engine contracts. A mode can run one engine directly (caveman or rtk) or a deterministic stacked pipeline that executes multiple engines in order.

Modes

Engine Registry

The registry lives in open-sse/services/compression/engines/registry.ts. Engines expose a shared contract:

• id: stable engine id such as caveman or rtk
• apply(text, config): legacy execution path used by stacked pipelines
• compress(input, config): primary execution path returning text + stats
• getConfigSchema(): returns the JSON-Schema-like shape of valid config
• validateConfig(config): returns { valid, errors[] }

Registration uses registerCompressionEngine(engine) (or registerEngine for advanced cases), which calls assertValidEngine() and validateConfig(defaultConfig) before accepting. Use unregisterCompressionEngine(id) to remove an engine at runtime.

strategySelector.ts registers the built-in engines before compression runs. This lets preview, runtime compression, stacked mode, tests, and future engines use the same execution path.

MCP description compression (related)

A separate registry compresses MCP tool description metadata at registry-level — see open-sse/mcp-server/descriptionCompressor.ts and MCP-SERVER.md. It reuses Caveman rules but operates on tool metadata, not request payloads.

Additional built-in engines

Beyond Caveman, RTK, and LLMLingua-2, the registry ships several specialized lossless / structural engines (used by stacked pipelines, the playground, and tests):

CCR retrieve-protocol instruction (#8033): the first time CCR replaces ≥1 block in a request, the engine prepends a single, idempotent system message (leading with the [CCR protocol] sentinel) teaching the caller the marker → tool contract: what a [CCR retrieve hash=<24hex> chars=N] marker means, that the hash must be copied verbatim (all 24 hex characters — mis-copied hashes are the likely cause of "block not found" misses), and that a [dedup:ref sha=...] marker means "look back in history", not "call the tool". The note is injected only when the caller's advertised tools[] proves it can actually reach omniroute_ccr_retrieve (callerSupportsCcrRetrieve() in open-sse/services/compression/engines/ccr/protocolInstruction.ts) — a plain OpenAI-compatible caller without that tool never receives an instruction to call something it cannot reach. Idempotency is enforced by scanning the message history for the sentinel before injecting, so multi-turn requests (which replay prior messages) do not stack the note once per turn.

Caveman

Caveman mode focuses on semantic condensation of normal prose:

• preserves code blocks, URLs, JSON, paths, and structured data
• removes filler, hedging, repeated context, and verbose connective phrasing
• supports language-aware file rule packs in open-sse/services/compression/rules/
• remains available through the legacy standard, aggressive, and ultra modes

The dashboard surface is Dashboard -> Context & Cache -> Caveman.

Caveman upstream reports ~75% fewer output tokens, 65% average output savings in benchmarks with a 22-87% range, and a ~46% input-compression tool. OmniRoute uses the Caveman input-side number when documenting stacked prompt/context savings; Caveman output mode remains a separate response-behavior feature.

RTK

RTK mode focuses on command and tool output:

• detects output classes such as git status, git branch, git diff, Vitest/Jest/Pytest, Cargo/Go tests, TypeScript/Vite/Webpack builds, ESLint, npm audit/installs, Docker logs, shell find/grep, stack traces, and generic logs
• applies 49 JSON filters from open-sse/services/compression/engines/rtk/filters/
• supports the RTK-style declarative pipeline: ANSI stripping, replace, match-output short-circuit, strip/keep lines, per-line truncation, head/tail/max-line truncation, and on-empty fallback
• supports trust-gated project filters in .rtk/filters.json and global filters in DATA_DIR/rtk/filters.json
• strips ANSI sequences, progress noise, repeated lines, and unhelpful boilerplate
• preserves actionable failures, warnings, summaries, changed files, and tail context
• can optionally retain redacted raw output for recovery/debugging through authenticated management routes

The dashboard surface is Dashboard -> Context & Cache -> RTK.

Operational details for custom filters, trust, verify, and raw-output recovery live in RTK_COMPRESSION.md.

RTK upstream reports 60-90% savings for command-output compression. Its README example shows a 30-minute Claude Code session going from ~118,000 tokens to ~23,900, or 79.7% saved.

LLMLingua-2 (Semantic Pruning)

LLMLingua-2 mode performs semantic token pruning on prose using a small ONNX token classifier, complementing the rule-based Caveman and RTK engines:

• compresses prose in non-system messages only; fenced code blocks and other preserved constructs are never altered
• runs the @atjsh/llmlingua-2 backend (ONNX via @huggingface/transformers) in a worker thread, so model inference never blocks the request event loop
• is stackable (stackPriority 35): in a stacked pipeline it runs after the structural engines (CCR, session-dedup, headroom, Caveman) but before ultra, since semantic pruning is most effective on already-structurally-compressed text — e.g. rtk -> caveman -> llmlingua
• fail-opens on any error (missing optional deps, worker spawn, model load, inference, or timeout) → the original text is returned unchanged, never an error

Engine location: open-sse/services/compression/engines/llmlingua/. The dashboard surface is Dashboard -> Context & Cache -> LLMLingua.

Models

The default model is TinyBERT (atjsh/llmlingua-2-js-tinybert-meetingbank, ~57 MB, fast). A higher-accuracy BERT-base model (Arcoldd/llmlingua4j-bert-base-onnx, ~710 MB) is available via the engine config model field. @huggingface/transformers downloads the selected model lazily from the HuggingFace Hub into ${DATA_DIR}/models/llmlingua on the first call (modelStore.ts); a modelPath config override points it at a local copy instead (offline / air-gapped installs).

Optional dependencies & on-demand install

The prunable LLMLingua runtime peer stack is optional. Three packages are declared as optionalDependencies in package.json and kept external by the production build (scripts/build/prepublish.ts does not bundle them):

@huggingface/transformers is pinned at 3.5.2 as an optional dependency (shared with the local embeddings path and also traced into the standalone bundle). Keeping it optional prevents onnxruntime-node CUDA provider postinstall failures on CUDA 11 hosts from aborting the whole OmniRoute install; when the optional stack is absent, LLMLingua still fail-opens. Only the three packages above are prunable SLM peers. A standard npm install (dev) installs the optional stack automatically unless optional dependencies are omitted.

Why on-demand: the npm-published package, the standalone bundle, and the Docker image ship without these deps to stay slim. When they are absent, the worker's dependency gate (a @atjsh/llmlingua-2 resolve probe in worker.ts) fails and the engine fail-opens silently — selecting LLMLingua becomes a no-op (text returned unchanged, no error logged). To activate it in a pruned environment, install the optional stack:

# pin to the versions declared in package.json optionalDependencies
npm install @atjsh/[email protected] @tensorflow/[email protected] js-tiktoken

Roughly ~800 MB total: the TensorFlow.js + transformers runtimes dominate; the TinyBERT model adds ~57 MB downloaded at first use (not via npm).

Per environment:

• Dev / npm install — installed automatically unless you passed --omit=optional (or --no-optional). No action needed.
• Global npm (npm i -g omniroute) / standalone — run the install command above inside the installed package directory, or reinstall without omitting optional deps.
• Docker — add the install command in a derived image layer; the published image ships slim by design.
• VPS (PM2) — install into the app's node_modules, then restart the process so the worker re-probes the gate.
• Raw Next standalone (npm run build → .build/next/standalone/server.js) — the standalone trace ships NEITHER the worker nor the optional deps, so the engine silently fail-opens. scripts/build/colocate-standalone.mjs re-applies both (worker esbuild + optional-dep closure into the standalone tree); it runs automatically via the postbuild npm hook after every build. Idempotent, fail-soft when deps are absent.

Verify it is active: with LLMLingua selected, real prose actually shrinks (the engine stops fail-opening), and the first request triggers the model download into ${DATA_DIR}/models/llmlingua. The gate intentionally probes only @atjsh/llmlingua-2 — the other peers are ESM-only and require.resolve throws on them even when present — so the worker still fail-opens if any peer is genuinely missing at import() time.

Stacked Pipelines

Stacked mode runs pipeline steps in order. The default is:

rtk -> caveman

Use this for coding-agent sessions where a prompt combines command output with human or assistant prose. RTK reduces noisy tool logs first, then Caveman compresses remaining natural language.

Pipeline steps are configured with stackedPipeline in compression settings or through compression combos.

When both engines reduce the same eligible payload, savings compound:

combined = 1 - (1 - RTK savings) * (1 - Caveman input savings)
average  = 1 - (1 - 0.80) * (1 - 0.46) = 89.2%
range    = 1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6%

MCP Accessibility Tree Filter

The MCP accessibility-tree smart filter is a post-execution compression layer that runs on MCP tool results, not on prompts or context. It targets the verbose accessibility-tree and browser snapshot payloads returned by tools like Playwright, computer-use, and browser-automation MCP servers.

What it does

1. Noise stripping — removes empty generic/text entries (- generic:, - text: "")
2. Sibling collapse — when ≥ collapseThreshold (default 30) consecutive lines are structural repeats, collapses them into the first collapseKeepHead (default 10) lines + a count summary + the last collapseKeepTail (default 5) lines
3. Ref preservation — [ref=eXX] anchors required by Playwright/computer-use are never touched
4. Hard truncation — if the text after collapse still exceeds maxTextChars (default 50,000), truncates with a navigation hint so the agent can continue working

Engine location

open-sse/services/compression/engines/mcpAccessibility/
  index.ts            ← smartFilterText() entry point
  collapseRepeated.ts ← sibling-collapse algorithm
  constants.ts        ← DEFAULT_MCP_ACCESSIBILITY_CONFIG

Configuration

Controlled by compression.mcpAccessibility in global settings (migration 056). Default config:

{
  "enabled": true,
  "maxTextChars": 50000,
  "collapseThreshold": 30,
  "collapseKeepHead": 10,
  "collapseKeepTail": 5,
  "minLengthToProcess": 2000
}

The filter is only applied to tool-result payloads whose type is "text" and whose length exceeds minLengthToProcess. It does not affect prompt compression or request payloads.

Expected savings

60–80% on browser snapshot tool results, depending on page complexity. The collapse algorithm is O(n) in line count and adds negligible latency.

This filter vs the compression engines above

Compression Combos

Compression combos are named compression profiles that can be assigned to routing combos:

• compression_combos: stores mode, pipeline, RTK config, language config, and default marker
• compression_combo_assignments: maps a compression combo to a routing combo
• runtime integration resolves an assigned compression combo before generic combo overrides
• analytics include compression_combo_id and engine

Dashboard surface: Dashboard -> Context & Cache -> Compression Combos.

API Surface

Management routes require management authentication or API-key policy checks.

MCP Tools

Compression exposes five MCP tools:

Scope & exclusions

Embeddings are never compressed. open-sse/handlers/embeddings.ts never calls any compression engine — the request/response bodies pass straight to the executor untouched. This is structural today (embeddings and chat completions are disjoint handlers), not a runtime check, but it means the vector-distortion concern in #8034 has no exposure surface in the embeddings path.

Per-model/endpoint exclusion filter (#8034). For chat completions, an operator can name model ids / provider/model targets that must never be compressed — a guardrail useful if compression is ever wired closer to an embeddings-adjacent path later, and generally useful for any model whose exact byte-for-byte prompt matters (deterministic evals, cache-sensitive prefixes, etc.).

• Settings field: exclusions?: string[] on the global compression config (GET/PUT /api/settings/compression), persisted via the existing key_value compression namespace (src/lib/db/compression.ts) — no new table.
• Dashboard tab: Dashboard → Compression → Exclusions (/dashboard/compression/exclusions).
• Pattern syntax: * is the only wildcard. Every other regex metacharacter in a pattern is escaped before matching, so gpt-5.6 matches the literal string only, never gpt-5x6 (ReDoS-safe, bounded, no nested quantifiers). Patterns match case-insensitively against both the bare model id and the provider/model composite — gpt-5-6, openai/gpt-5-6, and openai/* all work, and * alone excludes every model.
• Matching: isCompressionExcluded() / normalizeCompressionExclusions() in open-sse/services/compression/exclusions.ts. chatCore.ts checks the excluded target right after resolving compression settings, before any engine runs, and treats a match exactly like compression being globally disabled — the request body is provably byte-identical. The skip is recorded via writeCompressionSkip(..., "excluded") for analytics visibility.
• Default (empty/absent list): identical to pre-#8034 behavior — nothing is excluded.

Known limitations

• LLMLingua-2 (SLM) requires co-located optional deps. The worker only runs in a production build when @atjsh/llmlingua-2 + peers are co-located into dist/node_modules (see scripts/build/colocateOptionals.mjs, #4286). Without them the engine fail-opens (returns the original text). Worker resolution no longer depends on import.meta.url (it dies in the standalone bundle) — it anchors on the runtime cwd / argv[1].
• Caveman language packs de / fr / ja are partial. They ship context + filler + structural rules but no dedup / ultra packs, so ultra intensity is no stronger than full for those languages (they use only their own rules — there is no silent fall-back to the English dedup/ultra rules, which would mangle foreign text). en / es / id / pt-BR are complete. Contributions of dedup.json + ultra.json for the partial packs are welcome.
• Stacked telemetry only lists engines that compressed. A stacked-pipeline step whose engine ran but produced 0 % savings returns stats:null and so does not appear in engineBreakdown — indistinguishable from a step that was skipped. Distinguishing "ran, 0 %" from "skipped" would require a breakdown-model change and is deferred.

Validation

The focused gates for this area are:

node --import tsx/esm --test tests/unit/compression/rtk-*.test.ts tests/unit/compression/pipeline-integration.test.ts tests/unit/compression/context-compression-api.test.ts
node --import tsx/esm --test tests/unit/compression/*.test.ts tests/golden-set/*.test.ts tests/integration/compression-pipeline.test.ts tests/unit/api/compression/compression-api.test.ts
node --import tsx/esm --test tests/unit/compression/mcpAccessibility*.test.ts
npm run typecheck:core