Back to Airi

Verification: Flux Unbilled Exploit Fix

apps/server/docs/ai-context/verifications/flux-unbilled-exploit-fix.md

0.11.06.8 KB
Original Source

Verification: Flux Unbilled Exploit Fix

Status: patched in commit 7267b0d6b (2026-05-15) for the chat-completion path. TTS flux-meter adaptation followed up in the same PR as this verification doc (see "Remaining gaps → Gap 1"). Last attempted: 2026-05-15 Owner: [email protected]

用户路径

  • 场景:用户 0 < balance < fallbackRate,开 N 个并发 LLM completion 请求
  • 预期(修补后):第一个 request 之后 pre-flight 拒绝;最多触发一次 partial debit(charged < requested),所有剩余 request 收 402
  • 实际(修补前):N 个 request 全部走到 stream end,每个 catch consumeFluxForLLM 失败回滚 → balance 不变 → 用户拿到 N 次免费 LLM 响应

Before / After 行为

Before(修补前漏洞链)

  1. Pre-flight 仅 if (flux.flux <= 0)apps/server/src/routes/openai/v1/index.ts,旧版)
  2. 用户余额 1 flux,但 FLUX_PER_REQUEST (fallback) 通常远大于 1
  3. 并发 N 请求全部通过 pre-flight,上游 LLM 全部完成(响应已 stream 出去)
  4. consumeFluxForLLMdebitFlux (apps/server/src/services/billing/billing-service.ts:107),旧逻辑:
    ts
    if (balanceBefore < input.amount) {
      throw createPaymentRequiredError('Insufficient flux')
    }
    
    整个 DB tx 回滚 → balance 维持 1 flux
  5. catch 路径上报 airi_billing_flux_unbilled_total{reason='debit_failed'} += full amount
  6. 用户重复同样的脚本,每次都拿免费响应

这是 Grafana panel-43 (Flux Unbilled) 累积到 70.2K 的根因。

After(commit 7267b0d6b

1. Pre-flight 阈值改为 fallbackRateapps/server/src/routes/openai/v1/index.ts:138-142):

ts
const fallbackRate = await configKV.getOrThrow('FLUX_PER_REQUEST')
const fluxPer1kTokens = await configKV.get('FLUX_PER_1K_TOKENS')

const flux = await fluxService.getFlux(user.id)
if (flux.flux < fallbackRate) {
  throw createPaymentRequiredError('Insufficient flux')
}

并发 N 请求时,pre-flight 直接 402 拒绝,不会进入上游 LLM 调用。

2. Partial-debit 语义apps/server/src/services/billing/billing-service.ts:107-130):

ts
if (balanceBefore <= 0) {
  metrics?.fluxInsufficientBalance.add(1)
  throw createPaymentRequiredError('Insufficient flux')
}

const chargedAmount = Math.min(input.amount, balanceBefore)
const balanceAfter = balanceBefore - chargedAmount
const isPartial = chargedAmount < input.amount
  • balance > 0 但不够 → drain 到 0,ledger 记 amount = charged,metadata 带 requestedAmount + unbilled
  • balance <= 0 才 throw(catch path 上报 reason='debit_failed'
  • Partial drain 上报 reason='partial_debit_drained',跟真实 DB 错误区分

3. Idempotency 反映原始 chargedbilling-service.ts:71-94):

替换 request 的同 requestId 重放复用历史 existing.amount 作为 charged,避免重试时双扣 unbilled counter。

Evidence

ItemReference
Fix commit7267b0d6b feat(server/billing): partial-debit semantics to prevent unpaid usage exploit
Pre-flight gateapps/server/src/routes/openai/v1/index.ts:138-142
Partial-debit logicapps/server/src/services/billing/billing-service.ts:107-148
Streaming path unbilled metricapps/server/src/routes/openai/v1/index.ts:299-313 (label reason='partial_debit_drained', stage='streaming')
Non-streaming path unbilled metricapps/server/src/routes/openai/v1/index.ts:374-388 (label stage='non_streaming')
Regression tests addedapps/server/src/routes/openai/v1/route.test.ts (+97 lines), apps/server/src/services/billing/tests/billing-service.test.ts (+89 lines)

测试覆盖的两个核心 case:

  • 'rejects pre-flight when balance is below FLUX_PER_REQUEST (Issue: unpaid-usage-exploit)' — 验证 pre-flight 在 partial-balance 用户上 reject,upstream 未被调用
  • 'non-streaming completion drains partial balance and logs charged (Issue: unpaid-usage-exploit)' — 验证 partial-debit drain 到 zero + metric 上报

Remaining gaps

Gap 1 — TTS flux-meter partial-debit 适配 ✅ 已修复

apps/server/src/services/billing/flux-meter.ts:135-228accumulate() 现在解构 { charged, requested },partial drain 时:

  • 上报 airi_billing_flux_unbilled_total{source='tts_meter', reason='partial_debit_drained', meter, gen_ai.request.model?}
  • INCRBY (requested - charged) * unitsPerFlux 回 Redis debt counter
  • AccumulateResult 增加 unbilledFlux 字段供调用方观测
  • 加 invariant 校验 charged > requested / 非整数 / 负数 → throw 而不是静默 under-restore

测试覆盖:flux-meter.test.ts:217-260,case 名带 Issue: unpaid-usage-exploit follow-up

残余风险(已知,留 follow-up):settlement (LUA runScript) 和 INCRBY restore 之间非原子,并发请求理论上可能在窗口内读到 mid-state。窗口很小(一个 DB tx),实际命中很难触发;长期修复需要 Redis lock 或 unbilled 单独 bucket。代码里 flux-meter.ts// REVIEW: 标记。

Gap 2 — 修补前的 70.2K 历史漏账未核销

panel-43 显示的 70.2K 是 counter 累积值(increase($__range)),代表修补前漏出去的总量。修补不会让 panel 自动归零,只会让新的增量趋近 0(除真实 DB 错误)。

建议

  • 把 dashboard 时间窗调到 commit 7267b0d6b 部署后(2026-05-15 之后)观察增量斜率
  • 加 Grafana alert:increase(airi_billing_flux_unbilled_total[5m]) > 0 → PagerDuty
  • 如果业务需要核销历史 70.2K,从 flux_transaction ledger 反查 metadata->>'reason' = 'debit_failed' 的记录,配合 Loki 错误日志定位涉事 userId

What's verified

  • ✓ 代码层:pre-flight gate + partial-debit semantics 确实改了,逻辑正确
  • ✓ 测试层:两个核心 regression test 覆盖 exploit 场景,case 名带 Issue: unpaid-usage-exploit 标识

What's pending live verification

  • ⊘ 生产 Grafana 上 panel-43 在 2026-05-15 commit 部署后的斜率趋近 0
  • ⊘ 没跑 pnpm -F @proj-airi/server exec vitest run 实际确认新测试 pass(建议在 push 之前跑一次)
  • ⊘ TTS partial-debit 适配的 live verification(代码已修补 + 单测 14/14 pass,需要在生产观察 airi_billing_flux_unbilled_total{source='tts_meter'} 出现合理流量后再 close)

按优先级:

  1. P0 — 加 Grafana alert increase(airi_billing_flux_unbilled_total[5m]) > 0 → 通知
  2. P1 — 历史 70.2K 漏账处理:查 ledger + Loki 决定核销还是补账
  3. P1 — 修 TTS flux-meter 适配 partial-debit 新语义 ✅ 已修复,见 Gap 1
  4. P2 — Dashboard 改造(panel-43 时间窗注解 + 区分 reason label 的 stack 图,把 partial_debit_draineddebit_failed 分色展示)