Back to Codexbar

Spec: Contain Ultra-mode interleaved-lineage token overcounting (issue #2037)

docs/issue-2037-ultra-fork-overcount-spec.md

0.43.016.7 KB
Original Source

Spec: Contain Ultra-mode interleaved-lineage token overcounting (issue #2037)

  • Issue: steipete/CodexBar#2037 — "Ultra-mode Terra and Sol sessions can overcount forked context"
  • Status: Implemented (rev 4 — Phase 1 post-latch containment + min-cap)
  • Affected code: Sources/CodexBarCore/Vendored/CostUsage/ (Codex session scanner)
  • Related prior fixes: #968 (divergent totals), #1062 (repeated total snapshots), commit 45b68c34 (fork replay)

0. Framing (read first)

This spec deliberately ships in two phases:

  • Phase 1 (this PR): containment. Stop the multiplicative blowup and guarantee a never-inflates property. In multi-lineage files the result is an explicitly conservative estimate — it can undercount genuine totals-only sub-agent usage. We do not claim "true per-lineage usage."
  • Phase 2 (follow-up, fixture-gated): per-lineage accounting. Candidate-baseline run tracking to recover undercounted totals-only lineages. Blocked on obtaining a sanitized real Ultra fixture (§8), because its correctness depends on empirical last_token_usage semantics that cannot be asserted from first principles.

1. Problem

CodexBar massively overstates usage for gpt-5.6-terra and especially gpt-5.6-sol when they run in Ultra mode. In one reported Sol session, the raw log reached roughly 268M cumulative input tokens while CodexBar attributed 3.29B input tokens and about $4,000 of standard cost. A single forked turn contributed more than 3B tokens across hundreds of rows. Pricing tables are correct; the inflation comes from usage accounting.

Ultra sessions fork multiple sub-agents that:

  1. write interleaved cumulative token snapshots (total_token_usage) into the same session JSONL file, and
  2. replay large portions of the parent context into sub-agent turns.

2. Definitions

Lineage: one monotonic cumulative-counter sequence (total_token_usage) produced by one agent/sub-agent. Ultra files interleave several lineages with no reliable lineage identifier on token events (turn_id cannot be used: cumulative counters span turns in normal sessions).

Replay vs. genuine context — two different things this spec must not conflate:

  • Copied cumulative history: a child counter initialized with (or re-emitting) the parent's accumulated totals. Counting this again is double counting. Must always be excluded.
  • Context actually sent in a new sub-agent request: the replayed parent context is transmitted as input tokens of a real API call, typically billed at the cached-input rate. This may be genuine usage.

Which of these last_token_usage represents on a sub-agent's first turn in Ultra logs is an empirical question (§8). Phase 1 takes the conservative-for-inflation stance: after latch, last is capped by the contained totals delta (so below-watermark last is dropped). Phase 2 revisits this against the fixture.

3. Root cause

CostUsageScanner assumes one monotonic cumulative counter lineage per session file. handleTokenCount (in parseCodexFileCancellable, CostUsageScanner.swift) keeps a single rawTotalsBaseline and computes totals-derived deltas as max(0, current − baseline) via codexTotalDelta.

With two interleaved lineages A and B in one file the event stream looks like:

A: total=100M   → delta 100M, baseline=100M
B: total=5M     → clamped to 0 (divergent fallback), baseline=5M   ← baseline lowered
A: total=101M   → delta = 101M − 5M = 96M                          ← gap recounted
B: total=6M     → clamped to 0, baseline=6M
A: total=102M   → delta = 102M − 6M = 96M                          ← gap recounted again
...

Every lineage flip recounts nearly the entire gap between the two counters. Hundreds of interleaved snapshots inflate ~268M real tokens into billions.

Exposure by path:

  • Resolved fork child path (handleTokenCount, the forkedFromId != nil branch) runs totals-only — it deliberately ignores last to avoid replayed per-turn snapshots (45b68c34). Worst offender; matches "a single forked turn contributed 3B+".
  • Total-only events (no last_token_usage) hit the same gap-recount mechanism.
  • Root sessions with last present degrade differently: the divergent flag disables the #1062 guard (codexShouldPreferTotalDelta), so re-emitted snapshots can re-count last.

Existing mitigations do not cover this failure mode:

MitigationCoversGap
#968 divergent totals (codexDivergentTotalDelta)Counter decreases within one lineage → fall back to counted baselineLowers the effective baseline, so the next event of the larger lineage recounts the gap
#1062 (codexShouldPreferTotalDelta)Repeated identical total snapshots re-adding lastOnly active in the non-divergent path, and only compares adjacent totals; interleaving disables it and defeats adjacency (see §5.2)
Fork handling (forked_from_id, CodexInheritedTotalsResolver)Separate child files replaying parent historyNo concept of multiple lineages interleaved inside one file

4. Goals / non-goals

Goals (Phase 1)

  • Never-inflates invariant: for any input, tokens attributed from totals-derived deltas never exceed the maximum raw cumulative total observed in the file (fork-inheritance adjusted). No sequence of interleaved or re-emitted snapshots can multiply usage.
  • Exact re-emissions of previously seen snapshots contribute zero — including alternating re-emissions across lineages (this is where rev 1 of this spec was broken; see §5.2).
  • Fork-replayed parent history stays excluded (existing behavior preserved).
  • Single-lineage files behave exactly as today; all existing #968/#1062/fork tests pass unchanged.
  • Incremental (append-only) scans resume with all correctness-critical reducer state and produce byte-identical results to a full rescan; the optional seen-snapshot FIFO may be absent without affecting containment.
  • Multi-lineage results are explicitly documented as a conservative estimate (undercount bias), not exact usage.

Explicitly accepted Phase 1 limitation

A smaller lineage growing beneath another lineage's watermark (e.g. 5M → 50M under a 100M watermark) contributes nothing in Phase 1, even when it supplies last_token_usage: the contained totals delta is zero, so min(last, 0) = 0. This is normal Ultra behavior, not a rare edge case, and it is the deliberate trade: undercounting bounded genuine usage beats multiplying it. Phase 2 (§7) exists to recover it.

Non-goals

  • Per-sub-agent usage attribution/breakdown UI.
  • Changing pricing, priority/Ultra cost splitting (CostUsageScanner+CodexPriority.swift), or non-Codex providers.
  • Phase 2 candidate-run tracking (specified in outline only; separate PR).

5. Phase 1 design

All post-latch token-count accounting uses the shared tracker and delta helpers (§5.5), with correctness-critical state persisted for incremental scans (§5.6). Rules below are the shared policy, in precedence order.

5.1 High-watermark containment (load-bearing)

Track rawTotalsWatermark: the component-wise maximum of every raw cumulative total observed (after fork-inheritance adjustment).

  • Interleaving detection: an event whose total has any component strictly below the corresponding watermark component latches sawInterleavedTotals for the file (persisted, permanent). Mixed movement (input ↓ while output ↑) cannot come from one monotonic counter, so "any component" is deliberate. Legitimate single-lineage resets (compaction, restart, corrupt log) also latch the flag; that is accepted — it converts a potential overcount into an undercount.
  • Never lower the watermark or the baseline on detection.
  • Once latched, totals-derived deltas use codexContainedTotalDelta (§5.3.1), not a lowerable per-event baseline. Lineage flips cannot re-count the high/low gap.

Supersedes divergent mode: once sawInterleavedTotals is latched, codexDivergentTotalDelta (whose counted-baseline fallback is the gap-recount mechanism) and codexShouldPreferTotalDelta are not consulted for this file. sawDivergentTotals continues to work unchanged for never-interleaved files and for fork-parent snapshot resolution.

5.2 Seen-snapshot suppression (optional precision)

Maintain seenRawTotals: a bounded FIFO (~64) of raw cumulative totals for best-effort exact re-emission suppression.

  • Exact matches can short-circuit to zero before delta math.
  • After post-latch containment (§5.3), eviction cannot inflate usage: a re-emitted below-watermark total has contained delta 0, so min(last, 0) = 0.
  • Therefore the seen set is not load-bearing for correctness and must not gate incremental resume.

5.3 Counting rule in interleaved mode

For each token-count event once sawInterleavedTotals is latched and a total is present:

  1. Optionally, total exactly matches seenRawTotals → count 0 (precision optimization; not required for correctness).
  2. Compute the contained totals delta component-wise (§5.3.1).
  3. If last_token_usage is present → delta = min(adjustedLastDelta(last), containedTotalDelta).
  4. Otherwise → delta = containedTotalDelta.

last alone must never increase counted usage when the contained totals delta is zero. Smaller-lineage genuine last below the watermark is an accepted Phase 1 undercount.

5.3.1 Contained totals delta (not plain watermark delta)

Do not use plain codexTotalDelta(from: watermark, …) after latch — that breaks #968 “resume from counted baseline” (growth below the old raw watermark that still exceeds counted totals).

Use a dedicated helper, component-wise:

if current >= watermark {
    delta = max(0, current - max(watermark, counted))
} else {
    delta = max(0, current - counted)
}

This preserves counted-baseline recovery without allowing high/low lineage gaps to be recounted. Bound after latch:

counted ≤ max(counted_when_latched, subsequent_watermark) (and for totals-only streams, counted ≤ max observed cumulative total).

5.4 Fork children

  • Non-interleaved resolved forks keep totals-only accounting (45b68c34 / #1164). Do not apply a global min(last, total) cap there — established tests require totals-derived deltas that can exceed last.
  • After latch, resolved forks use the same post-latch rule as root sessions (§5.3), including min(adjustedLast, containedTotalDelta).
  • Unresolved forks keep skip-first + min(last, totalDelta); unresolvedForkTotalWatermark is a presence sentinel while the global tracker supplies the delta baseline.

5.5 Shared policy surface

CodexTotalsTracker (watermark + optional seen-set + latch) is shared. Post-latch delta policy (codexContainedTotalDelta / codexPostLatchEventDelta) must be applied by all three consumers:

  1. root / non-fork parsing (handleTokenCount)
  2. resolved-fork parsing (handleTokenCount)
  3. parent snapshot accumulation (CodexSnapshotAccumulator)

Otherwise fork children can inherit baselines computed under a different policy.

5.6 Cache: persist correctness-critical state

CostUsageFileUsage gains:

  • lastRawTotalsWatermark: CostUsageCodexTotals? (required for interleaved / divergent resume)
  • hasInterleavedTotals: Bool? (required when watermark is present; partial XOR → full rescan)
  • seenRawTotals: [CostUsageCodexTotals]? (optional precision only — missing must not force rescan)

Invalidation: regenerate CodexParserHash; clear compatibleCodexProducerKeys. Legacy divergent entries without a watermark force a per-file full rescan.

6. Acceptance criteria

  1. Exact expectations (primary): fixtures assert exact counted totals / row counts for each rule branch.
  2. Never-inflates property (merge bar): generated totals-only interleaved sequences satisfy counted ≤ max observed cumulative total.
  3. Floor: fixtures assert a conservative minimum so a degenerate ~0 parser fails.
  4. Manual / real Ultra log: no multiplied-gap inflation; satisfies the formal containment bound; above the fixture's conservative floor; plausible relative to the raw log — not required to land “near” 268M, because Phase 1 intentionally drops smaller-lineage usage.

7. Phase 2 outline (separate PR, fixture-gated)

Recover totals-only / below-watermark smaller-lineage usage with candidate-baseline run tracking, gated on a sanitized real Ultra fixture that establishes last_token_usage semantics.

8. Prerequisite: sanitized real Ultra fixture

Needed before claiming accurate multi-lineage recovery (Phase 2). Phase 1 ships on synthetic fixtures plus the containment property.

9. Touched files

FileChange
CostUsageScanner.swiftcodexContainedTotalDelta / codexPostLatchEventDelta; tracker + accumulator; post-latch policy in all three consumers
CostUsageScanner+CacheHelpers.swiftPersist watermark + interleaved flag; seen-set optional; incomplete-state rescan
CostUsageCache.swiftNew fields; clear compatible producer keys
CodexParserHash.generated.swiftRegenerated
CostUsageScannerBreakdownTests.swiftContainment / cap / eviction / cache / property tests
docs/issue-2037-ultra-fork-overcount-spec.mdThis spec

10. Test plan (merge bar)

  1. codex interleaved cumulative lineages do not recount the gap
  2. codex alternating repeated snapshots count zero — containment (not FIFO) keeps repeats at zero
  3. codex totals only growth below watermark is conservatively dropped
  4. codex single lineage counter reset undercounts but never inflates — preserves #968-style recovery past the peak
  5. codex interleaved fork child caps last by contained total delta
  6. codex root interleaved caps last much larger than watermark delta
  7. codex fork interleaved caps last much larger than watermark delta
  8. codex interleaved replay after sixty five unique snapshots stays contained
  9. codex interleaved totals only sequences stay within containment bound (property)
  10. codex incremental append preserves interleave containment across boundary — full state equality vs forceRescan
  11. codex missing watermark or interleaved flag forces full rescan
  12. codex missing optional seen set keeps incremental resume safe
  13. codex divergent cache entry without watermark forces full rescan
  14. Existing #968 / #1062 / fork replay tests unchanged

Regression: make check, focused scanner tests, then make test.


PR documentation (draft body — Phase 1)

Title

Contain Ultra-mode interleaved-lineage token overcounting (#2037)

Summary

  • Ultra-mode Terra/Sol sessions can interleave cumulative total_token_usage snapshots from multiple lineages in one JSONL file. A single file-global baseline then recounts the high/low gap on every flip — turning ~268M real input tokens into 3.29B ($4,000) in a reported session.
  • Phase 1 makes inflation provably bounded: a never-lower watermark latches interleaved mode on any component drop; post-latch deltas use a dedicated containment helper (preserving #968 counted-baseline recovery) and min(adjustedLast, containedTotalDelta) so last cannot grow usage when the contained totals delta is zero.
  • Single-lineage and pre-latch fork behavior is preserved (including #1164 totals-only fork replay handling). The slow JSON path and fork-parent snapshot builder share the same policy.
  • Cache persists watermark + interleaved flag (seenRawTotals is optional precision only); incomplete critical state forces a full rescan. Parser hash invalidation rebuilds old caches once.
  • Smaller-lineage undercount is intentional and deferred to fixture-gated Phase 2. Latched files are a conservative estimate, not claimed true per-lineage usage.

Fixes #2037. Related: #968, #1062, 45b68c34 (fork replay).

Behavior changes

  • Interleaved (Ultra) files: no multiplied-gap inflation; after latch, counted totals stay within the containment bound and last cannot exceed the contained totals delta.
  • Single-lineage / never-latched files: unchanged counting semantics.
  • Pre-latch resolved forks: still totals-only (replay-safe).
  • Post-latch: below-watermark smaller-lineage usage may be dropped until Phase 2.

Test plan

  • Focused CostUsageScannerBreakdownTests / CostUsageCacheTests (containment, caps, eviction, property bound, cache gates)
  • Existing #968 / #1062 / fork replay tests
  • make check
  • make test
  • Optional follow-up: rescan a real Sol/Ultra log for plausibility (not a merge blocker)

Notes for reviewers

  • Design is lineage-ID-free; value-based multi-run recovery is Phase 2 behind a sanitized Ultra fixture.
  • seenRawTotals is not load-bearing after the post-latch min-cap; missing it must not force a rescan.
  • A real Ultra log is valuable validation, not required to merge Phase 1 containment.