docs/issue-2037-ultra-fork-overcount-spec.md
Sources/CodexBarCore/Vendored/CostUsage/ (Codex session scanner)45b68c34 (fork replay)This spec deliberately ships in two phases:
last_token_usage semantics that cannot be asserted from first principles.CodexBar massively overstates usage for gpt-5.6-terra and especially gpt-5.6-sol when they run in Ultra mode. In one reported Sol session, the raw log reached roughly 268M cumulative input tokens while CodexBar attributed 3.29B input tokens and about $4,000 of standard cost. A single forked turn contributed more than 3B tokens across hundreds of rows. Pricing tables are correct; the inflation comes from usage accounting.
Ultra sessions fork multiple sub-agents that:
total_token_usage) into the same session JSONL file, andLineage: one monotonic cumulative-counter sequence (total_token_usage) produced by one agent/sub-agent. Ultra files interleave several lineages with no reliable lineage identifier on token events (turn_id cannot be used: cumulative counters span turns in normal sessions).
Replay vs. genuine context — two different things this spec must not conflate:
Which of these last_token_usage represents on a sub-agent's first turn in Ultra logs is an empirical question (§8). Phase 1 takes the conservative-for-inflation stance: after latch, last is capped by the contained totals delta (so below-watermark last is dropped). Phase 2 revisits this against the fixture.
CostUsageScanner assumes one monotonic cumulative counter lineage per session file. handleTokenCount (in parseCodexFileCancellable, CostUsageScanner.swift) keeps a single rawTotalsBaseline and computes totals-derived deltas as max(0, current − baseline) via codexTotalDelta.
With two interleaved lineages A and B in one file the event stream looks like:
A: total=100M → delta 100M, baseline=100M
B: total=5M → clamped to 0 (divergent fallback), baseline=5M ← baseline lowered
A: total=101M → delta = 101M − 5M = 96M ← gap recounted
B: total=6M → clamped to 0, baseline=6M
A: total=102M → delta = 102M − 6M = 96M ← gap recounted again
...
Every lineage flip recounts nearly the entire gap between the two counters. Hundreds of interleaved snapshots inflate ~268M real tokens into billions.
Exposure by path:
handleTokenCount, the forkedFromId != nil branch) runs totals-only — it deliberately ignores last to avoid replayed per-turn snapshots (45b68c34). Worst offender; matches "a single forked turn contributed 3B+".last_token_usage) hit the same gap-recount mechanism.last present degrade differently: the divergent flag disables the #1062 guard (codexShouldPreferTotalDelta), so re-emitted snapshots can re-count last.Existing mitigations do not cover this failure mode:
| Mitigation | Covers | Gap |
|---|---|---|
#968 divergent totals (codexDivergentTotalDelta) | Counter decreases within one lineage → fall back to counted baseline | Lowers the effective baseline, so the next event of the larger lineage recounts the gap |
#1062 (codexShouldPreferTotalDelta) | Repeated identical total snapshots re-adding last | Only active in the non-divergent path, and only compares adjacent totals; interleaving disables it and defeats adjacency (see §5.2) |
Fork handling (forked_from_id, CodexInheritedTotalsResolver) | Separate child files replaying parent history | No concept of multiple lineages interleaved inside one file |
Goals (Phase 1)
Explicitly accepted Phase 1 limitation
A smaller lineage growing beneath another lineage's watermark (e.g. 5M → 50M under a 100M watermark) contributes nothing in Phase 1, even when it supplies last_token_usage: the contained totals delta is zero, so min(last, 0) = 0. This is normal Ultra behavior, not a rare edge case, and it is the deliberate trade: undercounting bounded genuine usage beats multiplying it. Phase 2 (§7) exists to recover it.
Non-goals
CostUsageScanner+CodexPriority.swift), or non-Codex providers.All post-latch token-count accounting uses the shared tracker and delta helpers (§5.5), with correctness-critical state persisted for incremental scans (§5.6). Rules below are the shared policy, in precedence order.
Track rawTotalsWatermark: the component-wise maximum of every raw cumulative total observed (after fork-inheritance adjustment).
sawInterleavedTotals for the file (persisted, permanent). Mixed movement (input ↓ while output ↑) cannot come from one monotonic counter, so "any component" is deliberate. Legitimate single-lineage resets (compaction, restart, corrupt log) also latch the flag; that is accepted — it converts a potential overcount into an undercount.codexContainedTotalDelta (§5.3.1), not a lowerable per-event baseline. Lineage flips cannot re-count the high/low gap.Supersedes divergent mode: once sawInterleavedTotals is latched, codexDivergentTotalDelta (whose counted-baseline fallback is the gap-recount mechanism) and codexShouldPreferTotalDelta are not consulted for this file. sawDivergentTotals continues to work unchanged for never-interleaved files and for fork-parent snapshot resolution.
Maintain seenRawTotals: a bounded FIFO (~64) of raw cumulative totals for best-effort exact re-emission suppression.
min(last, 0) = 0.For each token-count event once sawInterleavedTotals is latched and a total is present:
seenRawTotals → count 0 (precision optimization; not required for correctness).last_token_usage is present → delta = min(adjustedLastDelta(last), containedTotalDelta).delta = containedTotalDelta.last alone must never increase counted usage when the contained totals delta is zero. Smaller-lineage genuine last below the watermark is an accepted Phase 1 undercount.
Do not use plain codexTotalDelta(from: watermark, …) after latch — that breaks #968 “resume from counted baseline” (growth below the old raw watermark that still exceeds counted totals).
Use a dedicated helper, component-wise:
if current >= watermark {
delta = max(0, current - max(watermark, counted))
} else {
delta = max(0, current - counted)
}
This preserves counted-baseline recovery without allowing high/low lineage gaps to be recounted. Bound after latch:
counted ≤ max(counted_when_latched, subsequent_watermark) (and for totals-only streams, counted ≤ max observed cumulative total).
45b68c34 / #1164). Do not apply a global min(last, total) cap there — established tests require totals-derived deltas that can exceed last.min(adjustedLast, containedTotalDelta).min(last, totalDelta); unresolvedForkTotalWatermark is a presence sentinel while the global tracker supplies the delta baseline.CodexTotalsTracker (watermark + optional seen-set + latch) is shared. Post-latch delta policy (codexContainedTotalDelta / codexPostLatchEventDelta) must be applied by all three consumers:
handleTokenCount)handleTokenCount)CodexSnapshotAccumulator)Otherwise fork children can inherit baselines computed under a different policy.
CostUsageFileUsage gains:
lastRawTotalsWatermark: CostUsageCodexTotals? (required for interleaved / divergent resume)hasInterleavedTotals: Bool? (required when watermark is present; partial XOR → full rescan)seenRawTotals: [CostUsageCodexTotals]? (optional precision only — missing must not force rescan)Invalidation: regenerate CodexParserHash; clear compatibleCodexProducerKeys. Legacy divergent entries without a watermark force a per-file full rescan.
counted ≤ max observed cumulative total.Recover totals-only / below-watermark smaller-lineage usage with candidate-baseline run tracking, gated on a sanitized real Ultra fixture that establishes last_token_usage semantics.
Needed before claiming accurate multi-lineage recovery (Phase 2). Phase 1 ships on synthetic fixtures plus the containment property.
| File | Change |
|---|---|
CostUsageScanner.swift | codexContainedTotalDelta / codexPostLatchEventDelta; tracker + accumulator; post-latch policy in all three consumers |
CostUsageScanner+CacheHelpers.swift | Persist watermark + interleaved flag; seen-set optional; incomplete-state rescan |
CostUsageCache.swift | New fields; clear compatible producer keys |
CodexParserHash.generated.swift | Regenerated |
CostUsageScannerBreakdownTests.swift | Containment / cap / eviction / cache / property tests |
docs/issue-2037-ultra-fork-overcount-spec.md | This spec |
codex interleaved cumulative lineages do not recount the gapcodex alternating repeated snapshots count zero — containment (not FIFO) keeps repeats at zerocodex totals only growth below watermark is conservatively droppedcodex single lineage counter reset undercounts but never inflates — preserves #968-style recovery past the peakcodex interleaved fork child caps last by contained total deltacodex root interleaved caps last much larger than watermark deltacodex fork interleaved caps last much larger than watermark deltacodex interleaved replay after sixty five unique snapshots stays containedcodex interleaved totals only sequences stay within containment bound (property)codex incremental append preserves interleave containment across boundary — full state equality vs forceRescancodex missing watermark or interleaved flag forces full rescancodex missing optional seen set keeps incremental resume safecodex divergent cache entry without watermark forces full rescan#968 / #1062 / fork replay tests unchangedRegression: make check, focused scanner tests, then make test.
Contain Ultra-mode interleaved-lineage token overcounting (#2037)
total_token_usage snapshots from multiple lineages in one JSONL file. A single file-global baseline then recounts the high/low gap on every flip — turning ~268M real input tokens into min(adjustedLast, containedTotalDelta) so last cannot grow usage when the contained totals delta is zero.seenRawTotals is optional precision only); incomplete critical state forces a full rescan. Parser hash invalidation rebuilds old caches once.Fixes #2037. Related: #968, #1062, 45b68c34 (fork replay).
last cannot exceed the contained totals delta.CostUsageScannerBreakdownTests / CostUsageCacheTests (containment, caps, eviction, property bound, cache gates)make checkmake testseenRawTotals is not load-bearing after the post-latch min-cap; missing it must not force a rescan.