Gate a rollout on traffic-weighted data, not a hand-picked cohort

Context

The plan for cutting the bootstrap public tier over to a new storage backend specified its verify gate up front as: "gate on the worst APAC cohort (hkg1/syd1/bom1/sin1) — the candidate's p99 must be materially below the incumbent there." Picking the worst-looking region and demanding the candidate win there feels like the conservative, rigorous choice.

It was neither. When the measurement window actually ran, that gate produced a table nobody could act on:

• APAC is a minority of the traffic. The user base was genuinely worldwide — the top countries by volume were the US, India, Italy, Australia, Germany, the UK, Japan, France, Singapore, and the UAE. Gating on four APAC cities decided a global cutover on a slice of it.
• One of the four named cities never gathered enough data to judge. Seoul finished the window with roughly 14–20 samples per side. It reported a "win" that meant nothing.
• A per-city p95 on a thin cell is noise, not a verdict. With ~20 samples, the 95th percentile is essentially the maximum observed value, so one slow read moves it arbitrarily.

Jakarta made the last point unmissable. Its p95 read 222 ms at n=4, then 1418 ms at n=32, then 1341 ms at n=56, then 799 ms at n=86 — a ~6× swing driven by sample count alone, before it settled. A gate evaluated at any single one of those moments would have returned a confident and different answer.

Guidance

Pick the gate's denominator from where users actually are. Replace "does the candidate win in cohort X" with a single traffic-weighted metric over all traffic — here, the share of all reads clearing the client latency budget. Weighting is automatic, no cohort has to be chosen, and the number is directly meaningful to the decision being made.

Then structure the gate as two separate tests, because they answer different questions:

1. Relative (the blocker): does the candidate regress anyone? Compare candidate vs incumbent per region, but only in cells with enough samples to judge. A regression in a well-sampled region blocks the rollout.
2. Absolute (the quality bar): what share of all traffic clears the target? Report it; hold it to a threshold. This is a bulk-quality measure, not a per-region hard requirement.

Add a minimum-sample threshold so thin cells report inconclusive rather than a false PASS or FAIL. This is the single highest-value line in a gate script. Without it, small cells emit confident verdicts that flip run to run.

Finally, keep the absolute bar from masquerading as a blocker. A region where the candidate is better than the incumbent yet still misses the absolute target is a follow-up, not a stop — because shipping still improves those users. Conflating the two is what made the first version of this gate hard-fail on a 32-sample remote city while the candidate was beating the incumbent there.

Why This Matters

Reframing from the APAC cohort to a global traffic-weighted metric did two things.

It produced a decisive answer where the cohort gate had produced an ambiguous one: the candidate cleared the budget for 99.6 % of all reads versus the incumbent's 84.4 % — the incumbent was missing roughly one request in six, globally.

More importantly, it surfaced the findings the cohort lens had hidden entirely:

• The incumbent was catastrophic in regions nobody had named as the worst case — clearing the budget for only ~40 % of reads in Africa and ~39 % in Oceania. The APAC-only gate would never have looked there.
• The incumbent's platform had no Middle East region at all, so MENA users were being served from Europe — while the candidate served them locally at ~270 ms with 100 % clearing the budget. For a MENA-centred user base this was arguably the single most valuable finding of the exercise, and the original gate could not have produced it.

The hand-picked cohort did not just measure the wrong thing — it pointed attention away from where the real problem was.

When to Apply

Apply whenever writing the accept/reject criteria for a cutover, migration, dependency swap, infrastructure change, or performance A/B — particularly when a plan or ticket already names a "worst case" cohort to gate on. Treat that named cohort as a diagnostic view, not the gate.

Also apply when reviewing a gate someone else wrote. The tells:

• The gate's denominator is a chosen subset rather than all traffic.
• Percentiles are computed per narrow cell with no sample-count guard.
• A single threshold serves as both "did anyone regress" and "is quality good enough".

Less relevant when the cohort genuinely is the population (a region-specific launch), or when per-cell volumes are large enough that thin-cell noise cannot arise.

Examples

Before — cohort gate with a hard absolute clause and no sample guard:

APAC = ["hkg1", "syd1", "bom1", "sin1"]
for city in APAC:
    ok = kv_p95[city] < redis_p95[city] and kv_p95[city] < BUDGET_MS
    if not ok:
        gate_pass = False          # a 32-sample city can hard-fail the whole rollout

After — traffic-weighted headline, sample-guarded relative test, separate quality bar:

MIN_SAMPLES  = 25    # below this a per-cell p95 sits on the near-max sample => inconclusive
BULK_TARGET  = 99.0  # % of ALL reads that must clear the budget

# 1. Relative test (the blocker) — only where the cell can actually be judged.
for city in regions:
    if kv_n[city] < MIN_SAMPLES or rds_n[city] < MIN_SAMPLES:
        verdict[city] = "thin"                      # inconclusive, NOT a pass or fail
    elif kv_p95[city] > rds_p95[city]:
        regressions.append(city)                    # this is what blocks
    elif kv_p95[city] > BUDGET_MS:
        budget_misses.append(city)                  # beats incumbent but misses target => follow-up

# 2. Absolute quality bar (traffic-weighted over ALL reads, no cohort chosen).
under_pct = 100.0 * reads_under_budget / reads_total

gate = (not regressions) and under_pct >= BULK_TARGET

The rewritten gate changed the reported outcome without any change in the underlying data: the same window that "FAIL"ed on a 32-sample remote city under the first version returned a clean pass with that city correctly reported as "beats incumbent, misses absolute target — follow-up", and the thin cell reported as inconclusive instead of a fabricated win.

Related

• docs/solutions/2026-07-16-bootstrap-kv-verify.md — the verify artifact this gate produced, including the continent-level table that the cohort framing would have omitted.
• docs/solutions/design-patterns/hedge-cache-read-against-origin-not-hard-timeout.md — the same window's data used for a different decision, and another instance of the general habit: measure the distribution before committing to a constant.