Compression Rules Format

Compression rules are JSON files loaded at runtime. They are intentionally data-only so new language packs and RTK command filters can be reviewed without changing engine code.

Canonical schema (source of truth): open-sse/services/compression/rules/_schema.json (JSON Schema draft 2020-12). The examples below are illustrative — when in doubt, validate your pack against _schema.json.

Caveman Rule Packs

Caveman rule packs live under:

open-sse/services/compression/rules/<language>/<pack>.json

Each pack contains replacements that apply to normal prose after protected regions are isolated.

{
  "language": "en",
  "category": "filler",
  "rules": [
    {
      "name": "question_to_directive",
      "pattern": "\\b(?:Can you explain why|Could you show me how)\\b\\s*",
      "replacement": "Explain why ",
      "replacementMap": {
        "can you explain why": "Explain why ",
        "could you show me how": "Show how "
      },
      "flags": "gi",
      "context": "all",
      "category": "context",
      "minIntensity": "lite",
      "description": "Convert verbose questions into direct requests."
    }
  ]
}

Caveman Fields

Use flags when case-sensitive matching matters, for example article removal before lowercase prose without stripping the OpenAI API. Use replacementMap when one regex has multiple alternatives that need different outputs; this keeps JSON rule packs data-only while preserving the behavior of the richer built-in TypeScript replacement functions.

RTK Filter Packs

RTK filters live under:

open-sse/services/compression/engines/rtk/filters/<filter>.json

Each filter describes how to recognize and compress a command-output family.

{
  "id": "test-vitest",
  "label": "Vitest output",
  "category": "test",
  "priority": 92,
  "match": {
    "outputTypes": ["test-vitest"],
    "commands": ["vitest", "npm test", "npm run test"],
    "patterns": ["\\bFAIL\\b", "\\bPASS\\b", "\\bTest Files\\b"]
  },
  "rules": {
    "stripAnsi": true,
    "replace": [{ "pattern": "\\s+\\[[0-9]+ms\\]", "replacement": "" }],
    "matchOutput": [
      { "pattern": "All tests passed", "message": "vitest: ok", "unless": "FAIL|Error:" }
    ],
    "includePatterns": ["FAIL", "Error:", "Test Files", "Tests"],
    "dropPatterns": ["^\\s*$", "Duration\\s+\\d+"],
    "collapsePatterns": ["^\\s+at "],
    "deduplicate": true,
    "truncateLineAt": 240,
    "maxLines": 160,
    "headLines": 24,
    "tailLines": 40,
    "onEmpty": "vitest: ok",
    "filterStderr": false
  },
  "preserve": {
    "errorPatterns": ["FAIL", "Error:", "AssertionError"],
    "summaryPatterns": ["Test Files", "Tests", "Snapshots"]
  },
  "tests": [
    {
      "name": "keeps failing tests",
      "command": "vitest",
      "input": "FAIL test/a.test.ts\\nError: boom\\nTest Files 1 failed",
      "expected": "FAIL test/a.test.ts\\nError: boom\\nTest Files 1 failed"
    }
  ]
}

RTK Fields

RTK applies declarative stages in this order: stripAnsi, filterStderr, replace, matchOutput, dropPatterns/includePatterns, truncateLineAt, headLines/tailLines, maxLines, and onEmpty.

Custom filters can be loaded from:

1. Project .rtk/filters.json files only after a matching .rtk/trust.json hash is present or trustProjectFilters is enabled.
2. Global DATA_DIR/rtk/filters.json.
3. Built-in filters.

Project/global custom files may contain one filter object or an array of filter objects. Invalid custom filters are skipped with diagnostics; invalid built-in filters fail validation.

Project trust file:

{
  "filtersSha256": "0123456789abcdef..."
}

The environment override OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=1 trusts project filters without a hash and should be limited to controlled local development.

Safety Rules

• Keep rules idempotent: running the same filter twice should not corrupt output.
• Preserve exact error text, file paths, line numbers, and command summaries where possible.
• Avoid rules that modify code blocks, JSON payloads, URLs, or secrets.
• Add unit coverage for new command families in detector/filter tests.
• Add tests[] samples to every built-in filter and to shared custom filters.

Validation

Rule packs are validated before use. Built-in Caveman packs and built-in RTK filters fail fast during validation so broken release assets are caught before shipment. Custom RTK filters are skipped with diagnostics when parsing or trust validation fails.

Focused validation:

node --import tsx/esm --test tests/unit/compression/rule-loader.test.ts tests/unit/compression/language-packs.test.ts
node --import tsx/esm --test tests/unit/compression/rtk-verify.test.ts tests/unit/compression/rtk-dsl-pipeline.test.ts