docs/design/2026-07-27-revert-pattern-triage-gate.md
Date: 2026-07-27
Status: Proposed
Area: CI triage — .github/workflows/qwen-triage.yml, .qwen/skills/triage/
Small behavior-neutral maintenance PRs currently consume the same multi-stage triage and model review capacity as behavior changes. The original proposal (PR #7414) attempted to filter these out, but a maintainer measurement of the live backlog showed only a ~2% hit rate — the feature targeted a near-nonexistent problem.
Meanwhile, the repo has 111 revert commits across its history (19 in July
2026 alone), and 61.5% of reverts occur within 24 hours of merge — meaning
the problem is caught quickly but after it's already on main. The real cost
is not reviewing harmless PRs; it's merging PRs that have to be rolled back.
This design proposes a data-backed triage gate that targets the PRs that actually cause reverts, not the ones that are already harmless.
Three-phase analysis of the repo's full revert history:
Collection: git log --all --grep="^Revert " found 111 revert commits.
Each revert's body was parsed for This reverts commit <hash>, then the
original commit was traced back to its PR via gh api. Result: 46 unique
reverted PRs (59 reverts traceable to a PR number; 52 reverts only had the
original commit title without a PR link).
Enrichment: For each reverted PR, extracted triage-time observable signals: touch scope (core/auth/providers/tools/services), diff size, review round count, bot Critical findings, CHANGES_REQUESTED cycles, merge→revert time gap, self-revert, E2E verification presence. 31 of 46 PRs were successfully enriched; 15 are deleted and inaccessible (HTTP 404).
Control group comparison: Sampled 60 recently-merged-but-not-reverted PRs and extracted the same signals. Computed precision (TP / (TP + FP)) and recall for each signal.
Scripts and raw data (local analysis artifacts, not committed):
.qwen/scripts/revert-analysis-*.mjs, .qwen/scripts/revert-data-*.json,
.qwen/scripts/revert-analysis-report-v2.json.
| Signal | Precision | Recall | Reverted (n=31) | Control (n=60) |
|---|---|---|---|---|
touches_high_risk | 66.7% | 32.3% | 10 | 5 |
non_maintainer + high_risk | 58.3% | 22.6% | 7 | 5 |
core + contested | 50.0% | 19.4% | 6 | 6 |
non_maintainer + core | 46.2% | 38.7% | 12 | 14 |
touches_core | 44.7% | 54.8% | 17 | 21 |
has_contested_pattern | 40.9% | 29.0% | 9 | 13 |
had_changes_requested | 40.7% | 35.5% | 11 | 16 |
non_maintainer | 39.6% | 67.7% | 21 | 32 |
large_diff_gt_200 | 37.0% | 54.8% | 17 | 29 |
critical_count > 0 | 28.6% | 12.9% | 4 | 10 |
fast_revert_24h | 100.0% | 25.8% | 8 | 0 |
self_reverted | 100.0% | 9.7% | 3 | 0 |
Sampling caveat: precision is computed on a 1:1.9 case-control ratio (31
reverted vs 60 control), while the repo's actual base rate is ~1.37% (46/3358).
Precision (PPV) is the metric most sensitive to this enrichment — the true
positive predictive value at the repo base rate is much lower (e.g. ~5% for
touches_high_risk). Sensitivity (recall) and specificity are invariant to
the sampling ratio and are the appropriate metrics for comparing signals.
The ranking of signals by precision is still valid (it is monotone in the
likelihood ratio at fixed n), but the absolute values should not be quoted
to contributors as posterior probabilities.
fast_revert_24h and self_reverted have 100% precision but are
post-merge signals — they cannot be used as triage gates because they are
only observable after the PR is already merged and reverted. They confirm the
problem exists but don't help prevent it.
critical_count > 0 was initially thought to be a strong signal (the bot
flagged the exact root cause in case studies like PR #6866), but after
correcting the regex to only match **[Critical]** tags (not the bare word
"critical" in prose like "no critical blockers"), the precision dropped to
28.6%. The bot is too trigger-happy with Critical findings — 16.7% of
control-group PRs also have Critical tags.
The touches_high_risk signal checks whether any changed file matches these
subsystem patterns:
openaiContentGenerator — streaming response parsingstreamingToolCallParser — tool call stream parsinggeminiChat — Gemini chat pipelineacpConnection — ACP process spawningshell.ts / shellExecutionService — shell tool executionmcp-client / mcp-pool — MCP server managementLspServer — LSP server managementacp-integration — ACP session integrationrelaunch.ts — desktop app relaunch lifecyclesandbox.ts — sandbox process managementelectron-run-as-node — Electron node-mode entry point (path match)These are the paths where incorrect changes are most likely to cause observable regressions that require reversion.
Of 13 PRs with valid (non-negative, post-merge) gap data:
This confirms that revert-causing defects surface quickly after merge, but
the damage is already on main.
8 PRs were reverted multiple times (revert → re-revert cycles), indicating unresolved contention:
These flip-flop PRs are the highest-cost outcomes — they consume multiple review rounds, multiple merge/revert cycles, and often require patch releases.
When a non-maintainer PR touches any high-risk path (see definition above),
Stage 1 triage escalates the PR to the deepest review tier instead of the
normal path. This does not block or close the PR — it ensures the full
/review pipeline runs with maximum agent coverage.
This is the strongest triage-time signal: 10 of 31 reverted PRs (32.3% sensitivity) touched these paths, vs 5 of 60 control PRs (91.7% specificity; Fisher p = 0.006).
Implementation: the Stage 1e skill text instructs the triage model to run
gh pr view --json files | grep -E '...' against the high-risk path patterns.
No workflow YAML change is needed — the detection runs inside the skill,
not as a separate workflow step.
large_diff_gt_200 has 37.0%
precision — size without context is not predictive.no_e2e is not
discriminative — 100% of the control group also lacks E2E comments, so
the signal cannot distinguish revert-prone PRs from safe ones.| PR #7414 (behavior-neutral) | This design (revert-pattern) | |
|---|---|---|
| Signal | "diff is entirely behavior-neutral" | "touches high-risk paths" |
| Revert recall | unmeasured (no reverts to compare) | 32.3% (10/31) |
| Specificity | n/a | 91.7% (55/60) |
| Targets | harmless PRs (cost: low) | dangerous PRs (cost: high) |
| False positive cost | skips review on a useful PR | escalates review depth (extra review time) |
.qwen/skills/triage/references/pr-workflow.md — add the Stage 1e high-risk
path checklist. The detection runs inside the triage skill (the model runs
gh api --paginate … | grep … itself),
so no workflow YAML change is needed.scripts/tests/qwen-triage-workflow.test.js — assert the high-risk path
routing strings exist in the triage skill markdown..github/scripts/qwen-triage-workflow.test.mjs — the same assertions in
the node:test runner.gh pr list API doesn't support deep pagination, so this requires a
GraphQL cursor-based fetch.main and alert maintainers.
This requires a separate monitoring workflow, not a triage gate.