docs/ops/QUALITY_GATE_PLAYBOOK.md
What this document is. A critical assessment of OmniRoute's quality-gate system, compared to industry best practices, plus a comprehensive catalog of all quality checkpoints and a tool-agnostic replication plan to apply the same system to any project. Generated on 2026-06-16 from the real repository state (not from memory).
Benchmarks: OWASP DSOMM · OpenSSF Scorecard · SLSA · SonarQube "Clean as You Code" · Quality-Ratchet pattern · DORA 2024 · OWASP LLM Top 10 (2025) · mutation-testing best practices.
For the gate-by-gate authoritative reference (what each gate validates, CI job, ratchet vs policy, blocking vs advisory), see the Quality Gates Reference.
Overall grade: A− / "Advanced". Top ~5–10% of projects. The system independently implements several patterns that the industry explicitly names — which is the strongest alignment signal (we didn't copy a checklist; we converged on the right practices).
| Reference framework | Where we stand | Grade |
|---|---|---|
| OWASP DSOMM (5 levels, 5 dimensions) | Solid Level 3, reaching 4 in Test Intensity and Static Depth. Most orgs sit at 1–2. | L3→L4 |
| OpenSSF Scorecard (18 checks) | We pass CI-Tests, Code-Review, Dependency-Update-Tool, Fuzzing, SAST, Signed-Releases (provenance), Token-Permissions, Vulnerabilities, Dangerous-Workflow. Gaps: Branch-Protection on main OFF; some actions not pinned. | ~7–8/10 |
| SLSA (4 levels) | npm publish --provenance + id-token: write + GitHub-hosted build = L2, approaching L3. Missing hardened/hermetic builder for L3+. | L2→L3 |
| SonarQube "Clean as You Code" | Identical philosophy: the ratchet gates non-regression (new code doesn't worsen the metric). Divergence: Sonar recommends few conditions; we have ~46 gates (fatigue risk). | Aligned, with caveat |
| Quality-Ratchet pattern | Reference implementation: ratchet + dedicatedGate + tightenSlack + --require-tighten + graceful-skip. More sophisticated than most public examples. | Exemplary |
| DORA 2024 | Very strong on stability axis. Risk: heavy gates can cost lead time — mitigated by fast-gates split, but with coverage gap (see Part 2). | Strong (stability) |
| OWASP LLM Top 10 (2025) | We cover risk #1 (prompt-injection) with runtime guard + promptfoo (eval) + garak (red-team). Standard industry tools. | Covered |
| Mutation testing | Stryker nightly, thresholds 70/50, 8 critical modules. Industry consensus (60% existing / 80% new, nightly) — we beat it. Gap: score is not yet a ratchet. | Almost there |
quality-baseline.json
up/down), tolerance (eps), slack
(tightenSlack) and dedicatedGate flag. Things that get fixed stay fixed — it's the
antidote to codebase entropy.useDefault) + SCA (osv/npm-audit/Trivy/Dependabot) + licenses + lockfile + SBOM + SLSA
provenance + Scorecard + workflow hardening (zizmor). Few codebases have this complete stack.check-test-masking (blocks weakening asserts to pass),
per-module coverage floors (forces testing HIGH-risk code, not just the easy parts) and
check-pr-evidence (Hard Rule #18).check-known-symbols,
check-fetch-targets, check-openapi-routes, check-docs-symbols ensure that docs, specs and
string dispatches point to living symbols. Catches "rot" that lint/test don't.--ratchet) exit exit 0 if the binary/network
fails — missing infra never blocks a legitimate PR. Mature engineering.trust-but-verify + stale-allowlist + evidence-gate
turn discipline into automated verification.quality.yml (PR→release/**)
now runs typecheck, fast deterministic tests, and an advisory production build for code PRs,
but it still does not run the full release-PR surface from ci.yml (coverage ratchets,
package artifact, integration, E2E, SonarQube). The motivation (speed) is valid, but the gate
should be where the merge happens (shift-left). Largest pending structural fix.osv (vulnCount) and oasdiff are
advisory despite frozen baselines. osv-advisory makes sense (a new CVE on an old dep would block
an unrelated PR) — but there's a middle ground (block only CRITICAL+fixable, as we did with
Trivy). oasdiff advisory means a contract-breaking change can pass.main is OFF. BRANCH_LOCK_TOKEN locks release branches, but
main itself is unprotected. Scorecard/DSOMM ding. Owner action required.codeql.yml gives more control; semgrep runs via an external cloud platform, not
versioned in the repo.The 12 categories below are the "quality system" in reusable form. Each lists the objective (what to protect), the tools we use and the tool-agnostic equivalent to replicate on any stack.
typecheck:core (blocking) + typecheck:noimplicit:core (advisory) + type-coverage ratchet 92.17% + per-file any-budget.any/escape-hatch budget.pr-test-policy (prod code requires a test), check-test-masking (blocks weakened asserts), pr-evidence (success claim requires evidence block), test-discovery (every test collected by a runner).[extend] useDefault=true — critical!), SonarQube, custom security rules (public-creds, error-helper, route-guard-membership, route-validation).check-deps anti-slopsquatting (allowlist + age ≥72h).--provenance), OpenSSF Scorecard (weekly), workflow hardening (zizmor: artipacked→persist-credentials:false, cache-poisoning, token-permissions).--threshold=65 + ratchet 80.1%).Build in phases, each delivering value on its own. Don't try all 12 categories at once — that causes exactly the gate fatigue Part 2 warns about. Every new gate enters advisory and becomes blocking when stable.
The entire system revolves around this 3-file pattern. Copy it first:
baseline.json — the frozen metric value + direction (up/down) + eps (anti-flake) + tightenSlack + dedicatedGate.collect-metrics.<ext> — runs the tool, extracts the number, writes metrics.json.check-ratchet.<ext> — compares metrics.json vs baseline.json; exit 1 only if regressed beyond eps; exit 0 (graceful skip) if the tool/infra was missing; with --require-tighten, exit 1 if it improved without updating the baseline (locks in the gain).With this in place, every new metric (coverage, complexity, warnings, SAST alerts, bundle size, mutation score…) is just one line in the baseline.
CI exists; formatter + linter + typecheck + 1 test runner + absolute coverage floor (e.g., 60%). Pre-commit runs fast auto-fixable checks. Output: no PR breaks the basics.
Implement the 3 files above. Freeze baselines for: warnings, coverage, complexity, duplication, dead code, file size. Output: the codebase can only improve from here.
SAST (CodeQL/Sonar/semgrep) with alert ratchet; secrets scanner (inherit the default ruleset); SCA (osv/Dependabot) + license allowlist + lockfile-lint. Output: known vulnerabilities and leaked secrets don't pass.
SBOM on publish + signed provenance (SLSA L2) + scheduled Scorecard + workflow hardening (zizmor: minimum tokens, no persisted credentials, pinned actions). Output: traceable and tamper-proof releases.
2nd runner if useful; per-module coverage floors for critical modules (anti-Goodhart);
property-based for pure logic; mutation testing nightly → when the 1st score arrives, make
mutationScore a ratchet. Output: coverage stops being a vanity metric; tests provably catch bugs.
If there's a public API: oasdiff (breaking-change, blocking) + schemathesis (nightly fuzz). DAST/red-team nightly as appropriate for the domain. Output: contracts don't break silently.
One consistency gate for each "duplicated truth" in the project. Domain-specific failure-mode gates (for AI: injection red-team). Output: structural rot and domain failures have a safety net.
stale-allowlist: every suppression has a justification + issue; obsolete suppression is caught.evidence-gate: success claim in a PR requires proof (test or living test).dedicatedGate for expensive metrics. Metrics that need an external binary get their own script (with skip), outside the synchronous central ratchet.P0 — highest ROI, almost ready
quality.yml production build after its
advisory week and keep moving deterministic release-PR-only checks into the PR→release path.main (owner setting) — boosts Scorecard, closes the DSOMM gap.P1 — valuable 4. osv/oasdiff → blocking with the right scope — osv only CRITICAL+fixable (two-step like Trivy); oasdiff blocks breaking-changes. 5. require-tighten → blocking (end of cycle) — locks in metric gains. 6. ROI/timing review per-gate in ci-summary — find and prune slow/low-value gates.
P2 — diminishing returns 7. SLSA L3 — hermetic/reproducible builder (GitHub SLSA generator) if you want to move up from L2. 8. Committed CodeQL config + versioned semgrep — more control/reproducibility. 9. Per-PR DAST smoke — fast subset of schemathesis/promptfoo on highest-risk endpoints (not just nightly). 10. Flakiness dashboard + DORA metrics — ensure gates aren't eroding speed.
This section records real incidents from release closures where a gate was missing, with concrete evidence and the proposed gate. Each item is a candidate for Part 5.
What happened. During the v3.8.27 /generate-release, the release PR (release/v3.8.27 → main)
was the first execution of the full ci.yml matrix in the integrated cycle. Result: 12 failures
at once — 3 deterministic tests + ~9 flakes/env. None were live product regressions, but
all went unnoticed because cycle PRs enter release/** via the Fast QG
(quality.yml), which does NOT run the full unit suite, nor pr-test-policy (test-masking), nor the
full integration suite, nor schema parity checking. The 3 deterministic ones:
permissions modal switch buttons declare button type:
#4034 added a 4th switch (a11y type="button" maintained); the test's === 3 count became
outdated. Static analysis should have caught this in the #4034 PR.findMissingArtifactPaths ... root runtime files:
dist/http-method-guard.cjs became a legitimate required-path; the test's expected list became
outdated.settings schemas accept ... unprefixed toggle: the modularized updateSettingsSchema (schemas/settings.ts, created by #3988) diverged
from the canonical one (settingsSchemas.ts): 45 fields vs 85 — 40 dropped + 6 divergent (qdrant*). It was
dead-code (runtime uses the canonical one), so no live impact, but only a hand-written parity
test caught it. #4030 restored 16 analogous drops from #3988/#3993, but this one slipped through.Proposed gates (Phase 9):
quality.yml (PR→release/**),
beyond typecheck + impacted tests, run pr-test-policy (test-masking) + the full deterministic
unit suite (or at least the static/parity files, which are fast and non-flaky).
This way, outdated tests and assert removal are caught in the PR that introduces them — not on
release day. Keep integration/e2e out (slow/flaky), but the deterministic layer CANNOT stay only
in PR→main.src/shared/validation/schemas/*, providerRegistry
modules, etc.), compares the shape (z.object keys, registry entries) against the canonical
source and fails on divergence (dropped/extra field). Would have caught the 40-field drop from
#3988 in that very PR. Generalizes the hand-written parity tests (which only exist where someone
remembered to write them). Cheap: imports both and diffs Object.keys(shape).known-flaky (quarantined with issue) so the release-PR red is only real signals, not noise
masking deterministic regressions in the middle.Principle: the gate has to run where the merge happens (already in "Cross-cutting principles"). The v3.8.27 incident shows this also applies to the deterministic test layer, not just lint/typecheck — otherwise the debt of outdated tests + lossy modularization only appears in PR→main, in batch, at the worst moment.