ComputerUse - Process Management for VNC Desktop Environment

This package provides a Computer Use plugin used by the Daytona Daemon to allow agents to control VNC desktop environments.

Overview

The ComputerUse package manages five main processes in the correct order:

1. Xvfb (X Virtual Framebuffer) - Provides a virtual display
2. xfce4 (Desktop Environment) - Starts the XFCE desktop environment
3. at-spi-bus-launcher (Accessibility bus) - Launches org.a11y.Bus so GTK/Qt/Electron apps publish their widget trees for the accessibility API
4. x11vnc (VNC Server) - Exposes the desktop via VNC protocol
5. novnc (Web-based VNC client) - Provides web access to the VNC server

Features

• Process Management: Automatic startup, monitoring, and shutdown of processes
• Priority-based Startup: Processes start in the correct order based on dependencies
• Auto-restart: Failed processes are automatically restarted
• Logging: Individual log files for each process
• Status Monitoring: Check the status of all processes
• Graceful Shutdown: Proper cleanup when stopping processes
• Individual Control: Start, stop, or restart individual processes

Dockerfile Requirements

To use the ComputerUse package, your Dockerfile must include the following VNC-related packages and setup:

Required Packages

# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive
ENV DISPLAY=:1
ENV VNC_PORT=5901
ENV NO_VNC_PORT=6080
ENV VNC_RESOLUTION=1280x720

# Install VNC and desktop environment packages
RUN apt-get update && apt-get install -y \
    wget \
    git \
    vim \
    xfce4 \
    xfce4-terminal \
    dbus-x11 \
    xfonts-base \
    xfonts-100dpi \
    xfonts-75dpi \
    xfonts-scalable \
    x11vnc \
    novnc \
    supervisor \
    net-tools \
    locales \
    xvfb \
    x11-utils \
    x11-xserver-utils \
    gnome-screenshot \
    scrot \
    imagemagick \
    xdotool \
    xautomation \
    wmctrl \
    build-essential \
    libx11-dev \
    libxext-dev \
    libxtst-dev \
    libxinerama-dev \
    libx11-xcb-dev \
    libxkbcommon-dev \
    libxkbcommon-x11-dev \
    libxcb-xkb-dev \
    libpng-dev \
    chromium \
    # AT-SPI (accessibility API)
    at-spi2-core \
    libatspi2.0-0 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

VNC Setup

# Setup VNC
RUN mkdir -p /home/daytona/.vnc && \
    chown -R daytona:daytona /home/daytona/.vnc

# NoVNC setup
RUN ln -sf /usr/share/novnc/vnc.html /usr/share/novnc/index.html && \
    sed -i 's/websockify =/websockify = --heartbeat 30/' /usr/share/novnc/utils/launch.sh

Launch Script

The NoVNC launch script (/usr/share/novnc/utils/launch.sh) is used to start the web-based VNC client. The Dockerfile modifies this script to add heartbeat support:

# Add heartbeat to websockify for better connection stability
sed -i 's/websockify =/websockify = --heartbeat 30/' /usr/share/novnc/utils/launch.sh

This ensures that the WebSocket connection remains stable during long-running sessions.

Additional Tools

The Dockerfile also installs several useful tools for VNC desktop interaction:

• xdotool: For mouse and keyboard automation
• xautomation: Additional automation tools
• wmctrl: Window manager control
• scrot: Screenshot capture
• imagemagick: Image processing
• gnome-screenshot: Alternative screenshot tool
• chromium: Web browser for testing

These tools are used by the toolbox API endpoints for desktop interaction.

Environment Variables

The following environment variables should be set in your Dockerfile:

ENV DEBIAN_FRONTEND=noninteractive
ENV DISPLAY=:1
ENV VNC_PORT=5901
ENV NO_VNC_PORT=6080
ENV VNC_RESOLUTION=1280x720
# AT-SPI accessibility API — ensures GTK/Qt apps publish their widget trees
ENV GTK_MODULES=gail:atk-bridge
ENV QT_ACCESSIBILITY=1
ENV NO_AT_BRIDGE=0

App-side accessibility caveats:

• GTK apps (xfce4, xfce4-terminal, Thunar, most of GNOME) publish their tree automatically once at-spi2-core is installed and GTK_MODULES=gail:atk-bridge is set. No per-app work.
• Qt apps need QT_ACCESSIBILITY=1 in their environment.
• Chromium and Electron need --force-renderer-accessibility on the command line. Without it they expose only a minimal AT-SPI subtree. This plugin does not enforce the flag — launch Chromium with it when you want agents to read browser DOM via AT-SPI, or use CDP instead.

User Setup

Ensure you have a non-root user with proper permissions:

# Create the Daytona user and configure sudo access
RUN useradd -m daytona && echo "daytona ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/91-daytona

# Switch to the user for VNC operations
USER daytona

Configuration

Environment Variables

• VNC_RESOLUTION: Set the VNC resolution (default: "1920x1080")
• VNC_PORT: VNC server port (default: 5901)
• NO_VNC_PORT: NoVNC web port (default: 6080)
• DISPLAY: X display (default: ":1")
• VNC_USER: User to run VNC processes (default: "daytona")

Process Configuration

The processes are configured with the following settings based on environment variables:

Default Values:

• DISPLAY: :1
• VNC_RESOLUTION: 1920x1080
• VNC_PORT: 5901
• NO_VNC_PORT: 6080
• VNC_USER: daytona

Log Files

Log files are stored in ~/.daytona/computeruse/:

• xfce4.log - Standard output from xfce4
• xfce4.err - Error output from xfce4

Integration with Toolbox

The ComputerUse package is integrated into the toolbox server and provides HTTP endpoints for:

• Screenshot functionality
• Mouse control
• Keyboard control
• Display information
• Accessibility (AT-SPI) — read the widget tree, query nodes by role/name/state, and act on them semantically

These endpoints are available under the /computeruse route group in the toolbox API.

Accessibility API (AT-SPI)

Agents can read the Linux desktop accessibility tree and act on nodes (focus, invoke, set value) without relying on screenshots or pixel coordinates. Useful for text-only agents and as a complement to vision for multimodal ones.

Endpoints

All under /computeruse/a11y/:

Node shape

{
  "id":          "<bus-name>:<object-path>",
  "role":        "push button",
  "name":        "Submit",
  "description": "",
  "bounds":      { "x": 120, "y": 340, "width": 96, "height": 32 },
  "states":      ["enabled","visible","showing","sensitive","focusable"],
  "actions":     ["press","click","release"],
  "children":    [ ... ]
}

children is present in tree responses and omitted in find results. Node IDs are AT-SPI accessible paths (D-Bus unique name + object path) — stable within a session, directly usable in subsequent action calls.

Filters (/find)

• scope: "focused" (default) | "pid" | "all". When pid, pass pid as an integer.
• role: exact, case-insensitive match against the AT-SPI role name.
• name + nameMatch: "exact", "substring" (default), or "regex". Substring and regex are case-sensitive.
• states[]: node must have all listed states.
• limit: max matches returned (default 500, ceiling 5000). Response includes "truncated": true when the cap is hit.
• All filter fields are AND-ed.

Error model

On 503, the rest of the computeruse API (mouse/keyboard/screenshot) stays up — agents can fall back to vision or pixel input.

App-side caveats

AT-SPI only reflects apps that publish their widget tree on the a11y bus:

• GTK apps (xfce4, xfce4-terminal, Thunar, most GNOME apps) — publish automatically once at-spi2-core is installed and GTK_MODULES=gail:atk-bridge is set.
• Qt apps — need QT_ACCESSIBILITY=1 in their environment.
• Chromium and Electron — need --force-renderer-accessibility on the command line. Without it they expose only a minimal subtree. This plugin does not enforce the flag; launch Chromium with it when you want AT-SPI coverage, or use CDP for browser automation instead.

What's not in v1

• No event subscriptions, streaming, or long-poll "wait until match" endpoint. Agents poll.
• No dedicated Table-interface endpoint (row/column coordinate lookup). Individual table cells that appear in the tree are reachable via /find + /node/* like any other node.
• No multi-select Selection-interface endpoints. Use focus + shift-click via the mouse endpoints.
• No drag-and-drop via AT-SPI — use the mouse drag endpoint.

Error Handling

The implementation includes comprehensive error handling:

• Process startup failures are logged and retried (if auto-restart is enabled)
• Log file access errors are handled gracefully
• Process termination is handled properly with context cancellation
• Mutex-based thread safety for concurrent access

Thread Safety

All operations are thread-safe using appropriate mutexes:

• sync.RWMutex for the main ComputerUse struct
• sync.Mutex for individual Process structs

This allows safe concurrent access from multiple goroutines.