docs/plans/2026-06-14-001-feat-doubao-asr-gateway-plan.md
Superseded direction as of 2026-06-14: do not use Volcengine AUC as AIRI's primary realtime ASR path. AUC requires URL-based recorded-file submission and is unsuitable for low-latency Hearing. The current implementation direction is to ship an official server-side realtime ASR proxy first, starting with Aliyun NLS because AIRI already has a working streaming transcription executor, then revisit Doubao streaming ASR (/api/v3/sauc/bigmodel_async) as a follow-up.
Add server-side official recorded-file transcription for AIRI through Doubao/Volcengine ASR. The client uploads a recorded audio file to AIRI, AIRI authenticates the user, stages the audio behind a short-lived public URL because Volcengine AUC accepts audio URLs, submits and polls the standard AUC API, maps the result back to an OpenAI-shaped transcription response, and bills successful requests through a new STT FluxMeter debt ledger.
This plan intentionally does not add realtime streaming ASR, client-side BYOK credentials, or a multi-provider ASR pool. The first user-facing path is recorded audio file transcription through the existing Hearing module and a new Official Transcription provider.
AIRI already has official hosted chat and TTS providers, plus client-side transcription providers for OpenAI, OpenAI-compatible endpoints, Aliyun NLS streaming, browser Web Speech, CometAPI, MiMo, and local audio paths. It does not yet have an official AIRI-hosted ASR provider that lets normal signed-in users transcribe recordings without bringing their own ASR credentials.
The server also does not expose a mounted transcription route today. The current AIRI audio surface is /api/v1/audio with speech, voices, and speech model catalog routes. The OpenAI public route surface under /api/v1/openai is kept for actual OpenAI-compatible chat endpoints, so ASR should extend the AIRI audio surface rather than adding another extension under /api/v1/openai.
Volcengine's recorded-file ASR APIs are asynchronous and require an online audio URL in the submit body. That means AIRI can present a normal multipart file upload to its clients, but the server needs a transient audio staging boundary before it can call the Doubao ASR upstream.
Product Behavior
file, model, optional language, optional response_format, and optional provider options.json and best-effort verbose_json responses compatible with the existing Hearing confidence filter. When upstream utterances are available, map them to segments with confidence and timing where possible.Server Gateway
POST /api/v1/audio/transcriptions, not under /api/v1/openai.Billing And Operations
POST /api/v1/audio/transcriptions beside /api/v1/audio/speech. This matches the current route split in apps/server/src/routes/openai/v1/index.ts where only actual OpenAI public endpoints stay under /api/v1/openai.https://www.volcengine.com/docs/6561/1354868 has submit and query endpoints intended for normal recorded-file recognition. The idle variant at https://www.volcengine.com/docs/6561/1840838 may complete within a 24h window, so it is not a good first fit for the synchronous Hearing settings test and recording workflow.LLM_ROUTER_CONFIG with ASR. Add an asr slice beside existing llm and tts models instead of creating a separate router config key. This reuses envelope key encryption, config cache invalidation, admin preview/apply semantics, model defaults, and router ownership.routeAsr rather than bypassing the router. ASR should become a first-class gateway operation, e.g. transcription.generate, with its own adapter contract and metrics. This keeps chat, TTS, and ASR diagnostics consistent.audio_info.duration. Do not trust a client-supplied duration for billing.X-Api-Key credentials for v1. The standard AUC docs support X-Api-Key. Start there, with resource id configured per model, and defer old-console X-Api-App-Key plus X-Api-Access-Key support unless operations needs it.flowchart TB
CLIENT[Stage Hearing module] --> PROVIDER[Official Transcription provider]
PROVIDER --> ROUTE[POST /api/v1/audio/transcriptions]
ROUTE --> GW[V1 gateway operation: transcription.generate]
GW --> PARSE[Parse multipart file and options]
PARSE --> STAGE[Stage audio to temporary public URL]
STAGE --> PREFLIGHT[STT Flux preflight by duration]
PREFLIGHT --> ROUTER[llmRouter.routeAsr]
ROUTER --> ADAPTER[Doubao ASR adapter]
ADAPTER --> SUBMIT[Volcengine AUC submit]
SUBMIT --> QUERY[Volcengine AUC query polling]
QUERY --> MAP[Map text, utterances, duration]
MAP --> BILL[sttMeter.accumulate]
BILL --> RESPONSE[OpenAI-shaped transcription response]
STAGE --> CLEANUP[Best-effort delete or TTL expiry]
sequenceDiagram
participant Client as Stage UI
participant Route as AIRI audio route
participant Staging as Audio staging
participant Router as LLM router ASR
participant Doubao as Volcengine AUC
participant Billing as STT FluxMeter
Client->>Route: multipart file, model auto, response_format
Route->>Staging: upload temporary object
Staging-->>Route: short-lived audio URL
Route->>Billing: assertCanAfford(duration seconds)
Route->>Router: routeAsr(model, audio URL, options)
Router->>Doubao: submit task
Doubao-->>Router: task id in response headers
loop bounded poll
Router->>Doubao: query task
Doubao-->>Router: processing, queued, success, or error
end
Router-->>Route: text, utterances, duration
Route->>Billing: accumulate(duration seconds)
Route-->>Client: json or verbose_json transcription
Route->>Staging: best-effort cleanup
apps/server/src/services/adapters/config-kv.tsapps/server/src/services/domain/llm-router/config-loader.tsapps/server/src/services/domain/llm-router/router.tsapps/server/src/services/domain/llm-router/types.tsapps/server/src/services/domain/llm-router/tests/router.test.tsLLM_ROUTER_CONFIG with an asr.models record. Add asrProviderSchema with a first provider value of volcengine-asr; use Doubao ASR as the user-facing/admin label. Add DEFAULT_ASR_MODEL, FLUX_PER_MINUTE_STT, and STT_DEBT_TTL_SECONDS ConfigKV entries. Add routeAsr and an AsrRouteContext that carries provider, model alias, upstream model/resource id, key entry id, timeout, and poll settings.LLM_ROUTER_CONFIG accepts llm, tts, and asr records with independent model ids.DEFAULT_ASR_MODEL makes model: "auto" fail with CONFIG_NOT_SET.routeAsr resolves model aliases, decrypts the configured key, and passes provider-specific adapter params without exposing ciphertext.apps/server/src/services/adapters/asr/types.tsapps/server/src/services/adapters/asr/volcengine.tsapps/server/src/services/adapters/asr/index.tsapps/server/src/services/adapters/asr/volcengine.test.tsapps/server/src/services/domain/llm-router/router.tsaudioUrl, file format, optional language, response format, upstream model name, and adapter params. Implement standard AUC submit/query using https://openspeech.bytedance.com/api/v3/auc/bigmodel/submit and /query by default, with endpoint overrides for tests and operations. Read status from the documented response headers and map success, processing, queued, silent audio, invalid request, empty audio, bad format, oversize, and busy states into AIRI gateway errors.fetch and exercise both header-level task status and JSON body mapping. Do not include real audio bytes in fixtures.X-Api-Key, resource id, request id, sequence header, model name, audio URL, and format.{ text, utterances, durationMs }.apps/server/src/services/domain/audio-staging/index.tsapps/server/src/services/domain/audio-staging/index.test.tsapps/server/src/app.tsapps/server/src/services/adapters/config-kv.tsapps/server/docs/ai-context/AudioStagingService interface with stage({ requestId, userId, file, contentType }) -> { url, objectKey, expiresAt } and cleanup(objectKey). Implement the first concrete backend only after choosing the deployment storage target. The repository does not currently show a server-side object storage/presigned URL boundary; unstorage appears only as a package dependency for packages/stage-ui, not as server upload infrastructure.apps/server/src/routes/openai/v1/index.tsapps/server/src/routes/openai/v1/gateway.tsapps/server/src/routes/openai/v1/types.tsapps/server/src/routes/openai/v1/operations/transcription-generation/index.tsapps/server/src/services/domain/openai-transcription/index.tsapps/server/src/routes/openai/v1/route.test.tsapps/server/src/app.tsapps/server/src/services/domain/openai-speech/index.ts. Parse multipart form data, resolve model: "auto" through DEFAULT_ASR_MODEL, stage the audio, derive trusted duration metadata for preflight, call llmRouter.routeAsr, map upstream result to OpenAI-style json or verbose_json, bill successful seconds through sttMeter, and emit request logs/product events/metrics. Add a route-specific upload limit so the global 1 MB body limit in app.ts does not silently reject normal audio files.model=auto routes to DEFAULT_ASR_MODEL and returns { text }.verbose_json request maps upstream utterances into segments and sets duration.sttMeter.accumulate.sttMeter.accumulate with ceil seconds derived from trusted duration.apps/server/src/app.tsapps/server/src/services/domain/billing/flux-meter.tsapps/server/src/services/domain/llm-tracing/index.tsapps/server/src/services/domain/product-events.tsapps/server/src/utils/observability.tsapps/server/docs/ai-context/flux-meter.mdapps/server/docs/ai-context/observability-conventions.mdsttMeter with service: "stt", FLUX_PER_MINUTE_STT, and STT_DEBT_TTL_SECONDS. Add tracing helpers such as startTranscriptionGeneration and OTel operation labels for transcription.generate. Record low-cardinality metrics by operation, provider, model, status, and duration bucket. Request/product logs can include request id, provider, model, file metadata, duration seconds, and status, but not raw audio or full transcript unless a deliberate transcript logging policy is added later.apps/server/src/routes/admin/config/router/index.tsapps/server/src/services/domain/admin/router-config/index.tsapps/server/src/services/domain/admin/router-config/index.test.tsapps/ui-admin/src/modules/api.tsapps/ui-admin/src/modules/router-config-form.tsapps/ui-admin/src/modules/router-config-form.test.tsapps/ui-admin/src/components/llm-router/RouterSliceEditor.vuevolcengine-asr labeled as Doubao ASR in the admin UI. It compiles to LLM_ROUTER_CONFIG.asr.models[modelName]. Expose fields for model alias, upstream model name, resource id, endpoint overrides, API key, key entry id, timeout/poll settings, and default ASR model. Preserve the existing preview/apply/redaction flow.LLM_ROUTER_CONFIG plus DEFAULT_ASR_MODEL as touched keys.packages/stage-ui/src/libs/providers/providers/official/index.tspackages/stage-ui/src/libs/providers/providers/official/shared.tspackages/stage-ui/src/composables/use-auth-provider-sync.tspackages/stage-ui/src/stores/providers.tspackages/stage-pages/src/pages/settings/providers/transcription/official-provider-transcription.vuepackages/i18n/src/locales/*/settings.yamlpackages/stage-ui/src/libs/providers/providers/official/index.test.tspackages/stage-ui/src/stores/modules/hearing.test.tsOFFICIAL_TRANSCRIPTION_PROVIDER_ID and providerOfficialTranscription using createOfficialAudioProvider() plus withCredentials(). Ensure the provider's transcription method posts through the existing @xsai/generate-transcription flow to /api/v1/audio/transcriptions. Add it to auth provider sync for the hearing module and provide auto model discovery if the server exposes an ASR model catalog, or a static auto model if it does not. Add a simple provider settings page with the existing transcription playground and no credential fields.x-airi-session-id.transcribeForRecording path and requests model: "auto".verbose_json confidence filtering works when the server returns segments and produces the current unsupported warning when it does not.apps/server/docs/ai-context/architecture-overview.mdapps/server/docs/ai-context/transport-and-routes.mdapps/server/docs/ai-context/flux-meter.mdapps/server/docs/ai-context/observability-conventions.mdapps/server/docs/ai-context/verifications/doubao-asr.md/api/v1/audio/transcriptions surface, document the temporary audio staging requirement, and add an env-guarded verification note for real Volcengine AUC tests./api/v1/openai/audio/speech while editing audio route docs.auto./api/v1/audio/transcriptions, AIRI stages it, Doubao returns text, and the UI displays the transcript.verbose_json segments so low-confidence text can be filtered by the existing Hearing store.LLM_ROUTER_CONFIG and DEFAULT_ASR_MODEL.| Surface | Impact |
|---|---|
apps/server routes | Adds POST /api/v1/audio/transcriptions and operation transcription.generate; route-specific upload limit must avoid the current global 1 MB body limit problem. |
apps/server router/config | Extends LLM_ROUTER_CONFIG with asr, adds ASR model defaults and adapter params. |
| Billing | Adds sttMeter using seconds/minutes and a sub-Flux debt ledger, parallel to TTS chars. |
| Observability | Adds ASR tracing, metrics, request logs, and product events without raw audio or transcript text by default. |
| Admin | Adds ASR slice support to router config preview/apply and UI builder modules. |
| Stage UI | Adds official transcription provider reused by stage-web, stage-tamagotchi/Electron, and stage-pocket/mobile through shared provider wiring. |
| Infrastructure | Requires temporary object storage with externally reachable URLs and a deletion/TTL policy. |
volc.seedasr.auc versus volc.bigasr.auc explicit.X-Api-Key, or must it also support old-console X-Api-App-Key plus X-Api-Access-Key credentials?apps/server/src/routes/openai/v1/index.ts defines the current route split: /api/v1/openai for OpenAI chat and /api/v1/audio for AIRI audio extensions.apps/server/src/routes/openai/v1/gateway.ts currently lists chat.completions and speech.generate; ASR needs a new operation.apps/server/src/services/domain/openai-speech/index.ts is the closest domain-service pattern for routing, tracing, billing, request logs, product events, and metrics.apps/server/src/services/adapters/config-kv.ts owns LLM_ROUTER_CONFIG, DEFAULT_TTS_MODEL, and TTS billing config; ASR config belongs near these definitions.apps/server/docs/ai-context/flux-meter.md and apps/server/docs/ai-context/billing-architecture.md already describe STT as a sub-Flux service category.packages/stage-ui/src/stores/modules/hearing.ts uses @xsai/generate-transcription for file-based transcription and already handles json/verbose_json.packages/stage-ui/src/libs/providers/providers/official/index.ts and shared.ts show how official providers attach AIRI auth and use /api/v1/audio.packages/stage-ui/src/composables/use-auth-provider-sync.ts is the auth-driven provider activation point that needs Hearing support.https://www.volcengine.com/docs/6561/1354868https://www.volcengine.com/docs/6561/1840838