sdk/runanywhere-swift/Sources/ONNXRuntime/README.md
The ONNXRuntime module provides speech-to-text (STT), text-to-speech (TTS), and voice activity detection (VAD) capabilities for the RunAnywhere Swift SDK using ONNX Runtime with models like Whisper, Piper, and Silero.
This module enables on-device voice processing with support for:
| Platform | Minimum Version |
|---|---|
| iOS | 17.5+ |
| macOS | 14.5+ |
The module requires:
RABackendONNX.xcframework (included in SDK)The ONNXRuntime module is included in the RunAnywhere SDK. Add it to your target:
dependencies: [
.package(url: "https://github.com/RunanywhereAI/runanywhere-sdks", from: "0.20.11")
],
targets: [
.target(
name: "YourApp",
dependencies: [
.product(name: "RunAnywhere", package: "runanywhere-sdks"),
.product(name: "RunAnywhereONNX", package: "runanywhere-sdks"),
]
)
]
https://github.com/RunanywhereAI/runanywhere-sdksRunAnywhereONNX to your targetRegister the module at app startup before using STT, TTS, or VAD capabilities:
import RunAnywhere
import ONNXRuntime
@main
struct MyApp: App {
init() {
Task { @MainActor in
ONNX.register()
try RunAnywhere.initialize(
apiKey: "<YOUR_API_KEY>",
baseURL: "https://api.runanywhere.ai",
environment: .production
)
}
}
var body: some Scene {
WindowGroup { ContentView() }
}
}
var req = RAModelLoadRequest()
req.modelID = "whisper-base-onnx"
req.category = .speechRecognition
req.framework = .onnx
let result = await RunAnywhere.loadModel(req)
print("Loaded: \(result.resolvedPath)")
let audioData: Data = // your audio data (16kHz, mono, Float32)
let output = try await RunAnywhere.transcribe(audio: audioData)
print("Transcribed: \(output.text)")
var options = RASTTOptions.defaults()
options.language = .en
options.sampleRate = 16000
options.enableWordTimestamps = true
let output = try await RunAnywhere.transcribe(audio: audioData, options: options)
print("Text: \(output.text)")
print("Confidence: \(output.confidence)")
if output.hasLanguageCode {
print("Detected language: \(output.languageCode)")
}
var options = RASTTOptions.defaults()
options.language = .en
for await partial in RunAnywhere.transcribeStream(audio: audioStream, options: options) {
if partial.isFinal {
print("Final: \(partial.text)")
} else {
print("Partial: \(partial.text)")
}
}
var unload = RAModelUnloadRequest()
unload.modelID = "whisper-base-onnx"
unload.category = .speechRecognition
_ = await RunAnywhere.unloadModel(unload)
var req = RAModelLoadRequest()
req.modelID = "piper-en-us-amy"
req.category = .speechSynthesis
req.framework = .onnx
_ = await RunAnywhere.loadModel(req)
var options = RATTSOptions.defaults()
options.rate = 1.0
options.pitch = 1.0
options.volume = 0.8
let output = try await RunAnywhere.synthesize(
"Hello! Welcome to RunAnywhere.",
options: options
)
// output.audioData contains the synthesized audio
// output.durationMs contains the audio length in ms
// Synthesize and play through device speakers
let result = try await RunAnywhere.speak("Hello world")
// With options
var options = RATTSOptions.defaults()
options.rate = 1.2
options.pitch = 1.0
let result = try await RunAnywhere.speak("Hello", options: options)
print("Duration: \(result.output.durationMs) ms")
for await chunk in RunAnywhere.synthesizeStream("Long text to synthesize...") {
// Process audio chunk as it is generated
playAudioChunk(chunk.audioData)
}
await RunAnywhere.stopSynthesis()
await RunAnywhere.stopSpeaking()
// Single buffer
var opts = RAVADOptions()
opts.sampleRate = 16000
opts.energyThreshold = 0.5
let samples: Data = // Float32 PCM at 16 kHz mono
let result = try await RunAnywhere.detectVoiceActivity(samples, options: opts)
if result.isSpeech {
print("Speech detected (confidence: \(result.confidence))")
}
// Stream VAD over a chunked AsyncStream<Data>
for await vadResult in RunAnywhere.streamVAD(audio: audioStream) {
if vadResult.isSpeech { handleSpeechFrame(vadResult) }
}
try await RunAnywhere.resetVAD()
public enum ONNX {
/// Module version
public static let version = "2.0.0"
/// Underlying ONNX Runtime version
public static let onnxRuntimeVersion = RAVersions.onnxRuntimeIOS // 1.24.3
/// Register the ONNX backend with the C++ service registry.
/// Registers the generic ONNX module (embeddings + Silero VAD) and the
/// Sherpa-ONNX engine plugin (STT: Whisper / Zipformer / Paraformer,
/// TTS: Piper / VITS) so `framework == .sherpa` resolves through the
/// unified C++ plugin router.
@MainActor
public static func register(priority: Int = 100)
/// Unregister the ONNX backend (also unregisters the Sherpa-ONNX plugin)
@MainActor
public static func unregister()
/// Trigger registration via property access (auto-registration helper)
public static let autoRegister: Void
}
ONNX is a thin public enum namespace. Model-to-backend routing (STT / TTS / VAD) is performed by the C++ plugin registry (rac_plugin_find / rac_plugin_find_for_engine) using the proto-typed RAInferenceFramework / RAModelCategory tables — there are no Swift-side canHandleSTT / canHandleTTS / canHandleVAD methods or capabilities set.
The ONNX module handles STT models containing:
whisper (Whisper variants)zipformer (Zipformer ASR)paraformer (Paraformer ASR)The ONNX module handles TTS models containing:
piper (Piper TTS voices)vits (VITS TTS models)The module uses Silero VAD by default for voice activity detection.
| Option | Type | Default | Description |
|---|---|---|---|
language | String | "en" | Language code for transcription |
sampleRate | Int | 16000 | Audio sample rate in Hz |
enableWordTimestamps | Bool | false | Include word-level timestamps |
enableVAD | Bool | true | Enable voice activity detection |
| Option | Type | Default | Description |
|---|---|---|---|
rate | Float | 1.0 | Speaking rate multiplier |
pitch | Float | 1.0 | Voice pitch multiplier |
volume | Float | 1.0 | Output volume (0.0 - 1.0) |
language | String | "en-US" | Voice language |
sampleRate | Int | 22050 | Output sample rate |
audioFormat | AudioFormat | .wav | Output audio format |
| Option | Type | Default | Description |
|---|---|---|---|
sampleRate | Int | 16000 | Audio sample rate in Hz |
frameLength | Double | 0.032 | Frame length in seconds |
energyThreshold | Double | 0.5 | Energy threshold for detection |
The module follows a thin wrapper pattern:
ONNX.swift (Swift wrapper)
|
ONNXBackend (C headers)
|
RABackendONNX.xcframework (C++ implementation)
|
+---------------+----------------+
| | |
ONNX Runtime Sherpa-ONNX Silero VAD
The Swift code registers the backend with the C++ service registry, which handles all model loading and inference operations internally.
| Device | Model | Real-time Factor |
|---|---|---|
| iPhone 15 Pro | Whisper Base | 0.3x (3x faster than real-time) |
| iPhone 15 Pro | Whisper Small | 0.5x |
| M1 MacBook | Whisper Base | 0.2x |
| M1 MacBook | Whisper Small | 0.3x |
| Device | Voice | Characters/sec |
|---|---|---|
| iPhone 15 Pro | Piper Amy | 200-300 |
| M1 MacBook | Piper Amy | 400-500 |
Performance varies based on model size and device thermal state.
RAModelInfo.isDownloadedregister() is called on the main actorregister() before RunAnywhere.initialize()Copyright 2025 RunAnywhere AI. All rights reserved.