Back to Eliza

Native Modules

packages/docs/apps/desktop/native-modules.md

2.0.120.9 KB
Original Source

The Eliza desktop app exposes platform capabilities to the web renderer through a set of native modules running in the Electrobun Bun host. Each module is initialized in initializeNativeModules(), and Bun-side request handlers are wired through registerRpcHandlers().

Renderer code talks to the host through window.__ELIZA_ELECTROBUN_RPC__:

  • request.<method>(params) for request/response calls into Bun
  • onMessage(name, listener) / offMessage(name, listener) for Bun push events

There are 10 native modules with 140+ request aliases and push messages in total, covering agent lifecycle, desktop integration, network discovery, voice I/O, wake-word detection, screen capture, camera, canvas windows, geolocation, and system permissions.

RPC Alias Conventions

The source of truth still keeps legacy channel-style aliases in rpc-schema.ts using the pattern <module>:<action> (for example agent:start, desktop:registerShortcut, gateway:startDiscovery). The public renderer API maps those aliases onto camelCase request methods and message names.

Examples:

  • agent:startwindow.__ELIZA_ELECTROBUN_RPC__.request.agentStart()
  • desktop:registerShortcutwindow.__ELIZA_ELECTROBUN_RPC__.request.desktopRegisterShortcut(...)
  • agent:status push event → window.__ELIZA_ELECTROBUN_RPC__.onMessage("agentStatusUpdate", ...)

Direction describes the communication model:

  • request — renderer calls Bun and awaits a response
  • message — Bun pushes an event to the renderer
typescript
const rpc = window.__ELIZA_ELECTROBUN_RPC__;

// Request: call a native module and await its response
const status = await rpc.request.agentStart();

// Message: listen for push notifications from the Bun host
rpc.onMessage("agentStatusUpdate", (status) => {
  console.log(status.state);
});

Modules

  • Agent — embedded Eliza runtime lifecycle
  • Desktop Manager — tray, shortcuts, windows, notifications, clipboard, shell
  • Gateway Discovery — mDNS/Bonjour local network scanning
  • Talk Mode — speech-to-text and text-to-speech pipeline
  • Swabble — wake-word detection
  • Screen Capture — screenshots and screen recording
  • Camera — camera enumeration, preview, and recording
  • Canvas — auxiliary BrowserWindow management and A2UI injection
  • Location — IP-based geolocation
  • Permissions — system permission checking and requesting

Agent

Class: AgentManager | Channels: 4 invoke, 1 event

Manages the embedded elizaOS agent runtime lifecycle — starting, stopping, restarting, and monitoring the local agent process.

Return typeAgentStatus:

typescript
interface AgentStatus {
  state: "not_started" | "starting" | "running" | "stopped" | "error";
  agentName?: string;
  port?: number;
  startedAt?: string;
  error?: string;
}
ChannelDirectionDescription
agent:startinvokeStarts the embedded agent. Returns AgentStatus.
agent:stopinvokeStops the running agent. Returns { ok: true }.
agent:restartinvokeStops and restarts the agent. Returns AgentStatus.
agent:statusinvokeReads the current agent state. Returns AgentStatus.
agent:statuseventPushed to the renderer whenever agent state changes. Payload: AgentStatus.

Desktop Manager

Class: DesktopManager | Channels: 32 invoke, 20 events

The largest native module. Wraps desktop system APIs for the tray, global shortcuts, auto-launch, window management, native notifications, power monitoring, clipboard, and shell operations.

Tray

ChannelDirectionDescription
desktop:createTrayinvokeCreates the system tray icon.
desktop:updateTrayinvokeUpdates the tray icon or tooltip.
desktop:destroyTrayinvokeRemoves the tray icon.
desktop:setTrayMenuinvokeSets the context menu items for the tray.
desktop:trayClickeventFired when the tray icon is left-clicked.
desktop:trayDoubleClickeventFired on a double-click of the tray icon.
desktop:trayRightClickeventFired on a right-click of the tray icon.
desktop:trayMenuClickeventFired when a tray menu item is selected. Payload includes the menu item id.

Global Shortcuts

ChannelDirectionDescription
desktop:registerShortcutinvokeRegisters a global keyboard shortcut.
desktop:unregisterShortcutinvokeUnregisters a specific shortcut by id.
desktop:unregisterAllShortcutsinvokeUnregisters all previously registered shortcuts.
desktop:isShortcutRegisteredinvokeReturns true if a shortcut id is currently registered.
desktop:shortcutPressedeventFired when a registered shortcut is triggered. Payload: { id: string }.

Auto Launch

ChannelDirectionDescription
desktop:setAutoLaunchinvokeEnables or disables launch at system startup.
desktop:getAutoLaunchStatusinvokeReturns whether auto-launch is currently enabled.

Window Management

ChannelDirectionDescription
desktop:setWindowOptionsinvokeApplies BrowserWindowConstructorOptions to the main window.
desktop:getWindowBoundsinvokeReturns { x, y, width, height } for the main window.
desktop:setWindowBoundsinvokeMoves and resizes the main window.
desktop:minimizeWindowinvokeMinimizes the main window.
desktop:maximizeWindowinvokeMaximizes the main window.
desktop:unmaximizeWindowinvokeRestores the main window from maximized state.
desktop:closeWindowinvokeCloses the main window.
desktop:showWindowinvokeShows a hidden main window.
desktop:hideWindowinvokeHides the main window without closing it.
desktop:focusWindowinvokeBrings the main window to the foreground.
desktop:isWindowMaximizedinvokeReturns true if the window is maximized.
desktop:isWindowMinimizedinvokeReturns true if the window is minimized.
desktop:isWindowVisibleinvokeReturns true if the window is visible.
desktop:isWindowFocusedinvokeReturns true if the window has focus.
desktop:setAlwaysOnTopinvokePins or unpins the window above all other windows.
desktop:setFullscreeninvokeEnters or exits fullscreen mode.
desktop:setOpacityinvokeSets the window opacity (0.0–1.0).
desktop:windowFocuseventFired when the main window gains focus.
desktop:windowBlureventFired when the main window loses focus.
desktop:windowMaximizeeventFired when the window is maximized.
desktop:windowUnmaximizeeventFired when the window is unmaximized.
desktop:windowMinimizeeventFired when the window is minimized.
desktop:windowRestoreeventFired when the window is restored from minimized state.
desktop:windowCloseeventFired when the window is closed.

Notifications

ChannelDirectionDescription
desktop:showNotificationinvokeDisplays a native OS notification.
desktop:closeNotificationinvokeDismisses a notification by id.
desktop:notificationClickeventFired when the user clicks a notification.
desktop:notificationActioneventFired when the user clicks an action button on a notification.
desktop:notificationReplyeventFired when the user submits a reply from a notification (macOS).

Power Monitoring

ChannelDirectionDescription
desktop:getPowerStateinvokeReturns AC vs battery plus live HID idle time and session lock state for LifeOps circadian inference: macOS (pmset -g batt + ioreg -c IOHIDSystem HID idle-time + CGSessionCopyCurrentDictionary lock state), Linux (/sys/class/power_supply Battery status), Windows (PowerStatus.PowerLineStatus). The idleTime (seconds) and idleState fields on DesktopPowerState are live on macOS; Linux/Windows still return them best-effort.
desktop:powerSuspendeventFired when the system is about to sleep.
desktop:powerResumeeventFired when the system wakes from sleep.
desktop:powerOnACeventFired when the system is plugged in.
desktop:powerOnBatteryeventFired when the system switches to battery.

App

ChannelDirectionDescription
desktop:quitinvokeQuits the desktop app.
desktop:relaunchinvokeRelaunches the app.
desktop:getVersioninvokeReturns the current app version string.
desktop:isPackagedinvokeReturns true if running a production build.
desktop:getPathinvokeReturns a desktop app path (for example userData or downloads).

Clipboard

ChannelDirectionDescription
desktop:writeToClipboardinvokeWrites text to the system clipboard.
desktop:readFromClipboardinvokeReturns the current clipboard text.
desktop:clearClipboardinvokeClears the clipboard.

Shell

ChannelDirectionDescription
desktop:openExternalinvokeOpens a URL in the default browser.
desktop:showItemInFolderinvokeReveals a file in Finder / Explorer.
desktop:beepinvokePlays the system beep sound.

Gateway Discovery

Class: GatewayDiscovery | Channels: 4 invoke, 1 event

Scans the local network for _eliza._tcp services using mDNS/Bonjour and surfaces discovered gateway instances to the renderer.

ChannelDirectionDescription
gateway:startDiscoveryinvokeStarts mDNS scanning on the local network.
gateway:stopDiscoveryinvokeStops the active scan.
gateway:getDiscoveredGatewaysinvokeReturns an array of currently known gateways.
gateway:isDiscoveringinvokeReturns true if a scan is in progress.
gateway:discoveryeventPushed when a gateway is found, updated, or lost. Payload: { type: "found" | "updated" | "lost", gateway }.

Talk Mode

Class: TalkModeManager | Channels: 10 invoke, 7 events

Manages the full speech pipeline: speech-to-text via Whisper or the Web Speech API, and text-to-speech via ElevenLabs or the system TTS engine.

ChannelDirectionDescription
talkmode:startinvokeStarts the Talk Mode session (begins listening).
talkmode:stopinvokeStops the active session.
talkmode:speakinvokeSends text to the TTS engine for playback.
talkmode:stopSpeakinginvokeCancels the current TTS playback.
talkmode:isSpeakinginvokeReturns true if TTS is actively playing.
talkmode:getStateinvokeReturns the current Talk Mode state object.
talkmode:isEnabledinvokeReturns true if Talk Mode is enabled in settings.
talkmode:updateConfiginvokeUpdates Talk Mode configuration at runtime.
talkmode:isWhisperAvailableinvokeReturns true if the local Whisper model is available.
talkmode:getWhisperInfoinvokeReturns metadata about the loaded Whisper model.
talkmode:transcripteventPushed when a speech-to-text transcript is ready.
talkmode:speakingeventPushed when TTS playback starts.
talkmode:speakCompleteeventPushed when TTS playback finishes.
talkmode:audioChunkeventPushed with raw PCM audio chunks during recording.
talkmode:audioCompleteeventPushed when audio recording ends.
talkmode:stateChangeeventPushed whenever the Talk Mode state changes.
talkmode:erroreventPushed when a speech pipeline error occurs.

Swabble

Class: SwabbleManager | Channels: 6 invoke, 3 events

Runs continuous wake-word detection in the background using fuzzy phrase matching. Whisper is used for transcription when available.

ChannelDirectionDescription
swabble:startinvokeStarts the wake-word listener.
swabble:stopinvokeStops the wake-word listener.
swabble:isListeninginvokeReturns true if the listener is active.
swabble:getConfiginvokeReturns the current wake-word configuration.
swabble:updateConfiginvokeUpdates the wake-word phrases and sensitivity at runtime.
swabble:isWhisperAvailableinvokeReturns true if Whisper is available for transcription.
swabble:stateChangeeventPushed when the listener starts or stops.
swabble:transcripteventPushed with the transcribed phrase that was detected.
swabble:wakeWordeventPushed when a configured wake-word is matched.

Screen Capture

Class: ScreenCaptureManager | Channels: 9 invoke, 1 event

Provides access to screen sources, screenshots, and screen recording.

  • App-window capture uses native OS tooling by default:
    • macOS: screencapture
    • Windows: PowerShell System.Drawing.CopyFromScreen
    • Linux: scrot, falling back to ImageMagick import
  • Game URL capture uses a dedicated BrowserWindow path when a game URL is provided.
ChannelDirectionDescription
screencapture:getSourcesinvokeReturns an array of available screen and window sources.
screencapture:takeScreenshotinvokeCaptures a full screenshot of the specified source.
screencapture:captureWindowinvokeCaptures a screenshot of a specific window by id.
screencapture:saveScreenshotinvokeSaves a screenshot buffer to disk and returns the file path.
screencapture:startRecordinginvokeStarts a screen recording session.
screencapture:stopRecordinginvokeStops the recording and returns the recorded file path.
screencapture:pauseRecordinginvokePauses an active recording session.
screencapture:resumeRecordinginvokeResumes a paused recording session.
screencapture:getRecordingStateinvokeReturns the current recording state (idle, recording, paused).
screencapture:recordingStateeventPushed whenever the recording state changes.

Camera

Class: CameraManager | Channels: 10 invoke, 0 events

Manages camera enumeration, live preview, photo capture, and video recording via a hidden renderer window. Permission checks are integrated directly into the module.

ChannelDirectionDescription
camera:getDevicesinvokeReturns a list of available camera devices.
camera:startPreviewinvokeStarts a live preview stream from the specified device.
camera:stopPreviewinvokeStops the active preview stream.
camera:switchCamerainvokeSwitches the active preview to a different camera device.
camera:capturePhotoinvokeCaptures a still photo from the current preview. Returns image data.
camera:startRecordinginvokeStarts video recording from the active camera.
camera:stopRecordinginvokeStops video recording and returns the file path.
camera:getRecordingStateinvokeReturns the current recording state.
camera:checkPermissionsinvokeReturns whether camera permission has been granted.
camera:requestPermissionsinvokePrompts the user for camera access permission.

Canvas

Class: CanvasManager | Channels: 16 invoke, 3 events

Creates and manages auxiliary BrowserWindow instances ("canvas windows") for headless or visible web navigation, JavaScript evaluation, snapshot capture, and A2UI message injection.

Window Lifecycle

ChannelDirectionDescription
canvas:createWindowinvokeCreates a new canvas window. Returns a window id.
canvas:destroyWindowinvokeDestroys a canvas window by id.
canvas:listWindowsinvokeReturns an array of all active canvas window ids and their current URLs.
ChannelDirectionDescription
canvas:navigateinvokeNavigates a canvas window to a URL.
canvas:evalinvokeEvaluates a JavaScript expression in a canvas window and returns the result.

Snapshots

ChannelDirectionDescription
canvas:snapshotinvokeCaptures a screenshot of a canvas window. Returns image data.

A2UI Messaging

ChannelDirectionDescription
canvas:a2uiPushinvokeInjects an A2UI message into the canvas window's renderer.
canvas:a2uiResetinvokeResets the A2UI state in a canvas window.

Visibility and Bounds

ChannelDirectionDescription
canvas:showinvokeMakes a canvas window visible.
canvas:hideinvokeHides a canvas window without destroying it.
canvas:resizeinvokeResizes a canvas window.
canvas:focusinvokeBrings a canvas window to the foreground.
canvas:getBoundsinvokeReturns { x, y, width, height } for a canvas window.
canvas:setBoundsinvokeMoves and resizes a canvas window.

Events

ChannelDirectionDescription
canvas:didFinishLoadeventPushed when a canvas window finishes loading a page.
canvas:didFailLoadeventPushed when a canvas window navigation fails. Payload includes the error code and description.
canvas:windowClosedeventPushed when a canvas window is closed (e.g., by a page calling window.close()).

Location

Class: LocationManager | Channels: 4 invoke, 2 events

Provides IP-based geolocation with position watching and local caching. Results are pushed as events for watched positions.

ChannelDirectionDescription
location:getCurrentPositioninvokeFetches the current geographic position. Returns a position object.
location:watchPositioninvokeStarts watching for position changes. Returns a watch id.
location:clearWatchinvokeStops watching a position by watch id.
location:getLastKnownLocationinvokeReturns the most recently cached location without a network call.
location:updateeventPushed when a watched position is updated. Payload: position object.
location:erroreventPushed when a geolocation error occurs. Payload: error details.

Permissions

Class: PermissionManager | Channels: 9 invoke, 1 event

Checks and requests OS-level permissions for accessibility, screen recording, microphone, camera, and shell access. Behavior varies by platform (macOS, Windows, Linux).

ChannelDirectionDescription
permissions:getAllinvokeReturns the current state of all tracked permissions.
permissions:checkinvokeChecks whether a specific permission is granted.
permissions:requestinvokePrompts the user to grant a specific permission.
permissions:openSettingsinvokeOpens the OS settings panel relevant to a permission.
permissions:checkFeatureinvokeReturns whether a named app feature is available given current permissions.
permissions:setShellEnabledinvokeEnables or disables shell access permission for the app.
permissions:isShellEnabledinvokeReturns whether shell access is currently enabled.
permissions:clearCacheinvokeClears the cached permission states, forcing a fresh check on next query.
permissions:getPlatforminvokeReturns the current platform identifier (darwin, win32, linux).
permissions:changedeventPushed when any permission state changes. Payload: updated permission map.

Usage Example

typescript
const rpc = window.__ELIZA_ELECTROBUN_RPC__;

// Register a global shortcut from the renderer
await rpc.request.desktopRegisterShortcut({
  id: "open-chat",
  accelerator: "CmdOrCtrl+Shift+M",
});

// Listen for the shortcut being pressed
rpc.onMessage("desktopShortcutPressed", ({ id }) => {
  if (id === "open-chat") openChatWindow();
});

// Start the embedded agent
const status = await rpc.request.agentStart();
console.log(status.state); // "starting" | "running" | "error"

// React to live agent state changes
rpc.onMessage("agentStatusUpdate", (nextStatus) => {
  updateAgentIndicator(nextStatus);
});

// Check a permission before using a feature
const cameraPermission = await rpc.request.permissionsCheck({ id: "camera" });
if (cameraPermission.status !== "granted") {
  await rpc.request.permissionsRequest({ id: "camera" });
}

The renderer usually should not call this global directly. Prefer the shared app helpers in eliza/packages/app-core/src/bridge/electrobun-rpc.ts, which wrap the same preload bridge behind request/message helpers used by the app and plugins.


  • Desktop App — overview of the desktop application architecture
  • Deep Linkingeliza:// URL protocol handled via the Canvas module
  • Capacitor Plugins — equivalent native plugin system for iOS and Android