Hardware Validation

Live Device Testing (AI Agents)

PRIMARY TOOL: bash validate - Hardware-in-the-loop validation framework:

The bash validate command is the preferred entry point for AI agents doing live device testing. It provides a complete validation framework with pre-configured expect/fail patterns designed for hardware testing.

MANDATORY: Driver Selection Required

You must specify at least one LED driver to test using one of these flags:

--parlio - Test parallel I/O driver
--rmt - Test RMT (Remote Control) driver
--spi - Test SPI driver
--uart - Test UART driver
--i2s - Test I2S LCD_CAM driver (ESP32-S3 only)
--lcd-rgb - Test LCD RGB driver (ESP32-P4 only)
--all - Test all drivers (equivalent to --parlio --rmt --spi --uart --i2s --lcd-rgb)
--parallel - Test multiple drivers simultaneously (requires 2+ drivers)

Usage Examples

bash

# Test specific driver (MANDATORY - must specify at least one)
bash validate --parlio                    # Auto-detect environment
bash validate esp32s3 --parlio            # Specify esp32s3 environment
bash validate --rmt
bash validate --spi
bash validate --uart
bash validate --i2s                       # ESP32-S3 only
bash validate --lcd-rgb                   # ESP32-P4 only

# Test multiple drivers (sequentially)
bash validate --parlio --rmt              # Auto-detect environment
bash validate esp32dev --parlio --rmt     # Specify esp32dev environment
bash validate --spi --uart

# Test multiple drivers in parallel (simultaneously on device)
bash validate --parlio --lcd-rgb --parallel --lanes 1    # PARLIO + LCD_RGB parallel on ESP32-P4
bash validate --parlio --rmt --parallel                  # PARLIO + RMT parallel

# Test all drivers
bash validate --all                       # Auto-detect environment
bash validate esp32s3 --all               # Specify esp32s3 environment

# Combined with other options
bash validate --parlio --skip-lint
bash validate esp32s3 --rmt --timeout 120
bash validate --all --skip-lint --timeout 180

Common Options

--skip-lint - Skip linting for faster iteration
--timeout <seconds> - Custom timeout (default: 120s)
--help - See all options

Strip Size Configuration

Configure LED strip sizes for validation testing via JSON-RPC:

bash

# Use strip size presets
bash validate --parlio --strip-sizes small         # 100, 500 LEDs
bash validate --rmt --strip-sizes medium           # 300, 1000 LEDs
bash validate --spi --strip-sizes large            # 500, 3000 LEDs

# Use custom strip sizes (comma-separated LED counts)
bash validate --parlio --strip-sizes 100,300       # Test with 100 and 300 LED strips
bash validate --rmt --strip-sizes 100,300,1000     # Test with 100, 300, and 1000 LED strips
bash validate --i2s --strip-sizes 500              # Test with single 500 LED strip

# Combined configuration
bash validate --all --strip-sizes 100,500,3000

Strip Size Presets:

tiny - 10, 100 LEDs
small - 100, 500 LEDs (default)
medium - 300, 1000 LEDs
large - 500, 3000 LEDs
xlarge - 1000, 5000 LEDs (high-memory devices only)

Lane Configuration

Configure number of lanes for validation testing via JSON-RPC:

bash

# Test with specific lane count
bash validate --parlio --lanes 2                   # Test with exactly 2 lanes
bash validate --i2s --lanes 4                      # Test with exactly 4 lanes

# Test with lane range
bash validate --rmt --lanes 1-4                    # Test with 1 to 4 lanes (tests all combinations)
bash validate --spi --lanes 2-8                    # Test with 2 to 8 lanes

# Set per-lane LED counts (NEW)
bash validate --i2s --lane-counts 100,200,300      # 3 lanes with 100, 200, 300 LEDs per lane
bash validate --parlio --lane-counts 50,100        # 2 lanes with 50 and 100 LEDs per lane

# Combined with strip sizes
bash validate --i2s --lanes 2 --strip-sizes 100,300  # 2 lanes, strips of 100 and 300 LEDs

Default: 1-8 lanes (firmware default)

Color Pattern Configuration

Configure custom RGB color pattern for validation testing:

bash

# Custom color patterns (hex RGB format)
bash validate --parlio --color-pattern ff00aa      # Pink color (RGB: 255, 0, 170)
bash validate --rmt --color-pattern 0x00ff00       # Green color (RGB: 0, 255, 0)
bash validate --i2s --color-pattern 112233         # Dark blue (RGB: 17, 34, 51)

# Combined with lane configuration
bash validate --parlio --lane-counts 100,200 --color-pattern ff0000  # 2 lanes, red color

Note: Custom color patterns require firmware support via the setSolidColor RPC command. This command may need to be implemented in the firmware if not already available.

Error Handling

If you run bash validate without specifying a driver, you'll get a helpful error message:

ERROR: No LED driver specified. You must specify at least one driver to test.

Available driver options:
  --parlio    Test parallel I/O driver
  --rmt       Test RMT (Remote Control) driver
  --spi       Test SPI driver
  --uart      Test UART driver
  --i2s       Test I2S LCD_CAM driver (ESP32-S3 only)
  --lcd-rgb   Test LCD RGB driver (ESP32-P4 only)
  --all       Test all drivers

Example commands:
  bash validate --parlio
  bash validate esp32s3 --parlio
  bash validate --rmt --spi
  bash validate --all

What bash validate Does

Lint - Catches ISR errors and code quality issues (skippable with --skip-lint)
Compile - Builds for attached device
Upload - Flashes firmware to device
Monitor - Validates output with smart pattern matching
Report - Exit 0 (success) or 1 (failure) with detailed feedback

Runtime Driver Selection

Driver selection happens at runtime via JSON-RPC (no recompilation needed). You can instantly switch between drivers or test multiple drivers without rebuilding firmware.

Why bash validate instead of bash debug

Pre-configured patterns catch hardware failures automatically
Designed for automated testing and AI feedback loops
Prevents false positives from ISR hangs and device crashes
Comprehensive test matrix (48 test cases across drivers/lanes/sizes)
Runtime driver selection (no recompilation needed)

When to use bash debug (advanced): For custom sketches requiring manual pattern configuration. See Device Debug section below.

Device Debug (Manual Testing)

bash debug <sketch> - Low-level tool for custom device workflows:

Simple example:

bash

bash debug MySketch --expect "READY"

Complex example:

bash

bash debug MySketch --expect "INIT OK" --expect "TEST PASS" --fail-on "PANIC" --timeout 2m

For full documentation, see uv run ci/debug_attached.py --help

JSON-RPC Validation Protocol (Advanced)

JSON-RPC AS A SCRIPTING LANGUAGE: The validation system uses JSON-RPC as a scripting language for hardware testing with fail-fast semantics:

Fail-Fast Model

All test orchestration happens via JSON-RPC commands and responses
Text output (FL_PRINT, FL_WARN) is for human diagnostics ONLY
Python code never greps serial output for control flow decisions
Tests exit immediately on pass/fail (no timeout waiting)

Example JSON-RPC Test Script

python

from ci.rpc_client import RpcClient

with RpcClient("/dev/ttyUSB0") as client:
    # Pre-flight check
    result = client.send("testGpioConnection", args=[1, 2])
    if not result["connected"]:
        exit(1)  # Fail-fast: hardware not connected

    # Configure and run test in single call (NEW consolidated format)
    result = client.send("runTest", args={
        "drivers": ["PARLIO", "RMT"],
        "laneRange": {"min": 1, "max": 4},
        "stripSizes": [100, 300]
    })
    exit(0 if result["success"] else 1)

EXTENSIBLE TESTING

The validation system uses bidirectional JSON-RPC over serial for hardware-in-the-loop testing. This protocol enables AI agents to:

Test LED drivers without recompilation (runtime driver selection)
Add new test functionality by extending the JSON-RPC API
Configure variable lane sizes and test patterns programmatically

Python Agent (`ci/validation_agent.py`)

python

from validation_agent import ValidationAgent, TestConfig

# Connect to device
with ValidationAgent("COM18") as agent:
    # Health check
    agent.ping()  # Returns: {timestamp, uptimeMs, frameCounter}

    # Get available drivers
    drivers = agent.get_drivers()  # Returns: ["PARLIO", "RMT", "SPI", "UART"]

    # Configure test: 100 LEDs, 1 lane, PARLIO driver
    config = TestConfig(
        driver="PARLIO",
        lane_sizes=[100],  # Per-lane LED counts
        pattern="MSB_LSB_A",
        iterations=1
    )
    agent.configure(config)

    # Run test and get results
    result = agent.run_test()
    print(f"Passed: {result.passed_tests}/{result.total_tests}")

    # Reset between tests
    agent.reset()

JSON-RPC Commands

Sent over serial with REMOTE: response prefix:

Command	Args	Description
`ping`	-	Health check, returns timestamp and uptime
`drivers`	-	List available LED drivers with enabled status
`getState`	-	Query current device state and configuration
`runTest`	`{drivers, laneRange?, stripSizes?}`	NEW: Configure and execute test in single call with named arguments (recommended)
`setDrivers`	`[driver_names...]`	Legacy: Set which drivers to test (use `runTest` with config instead)
`setLaneRange`	`min, max`	Legacy: Set lane count range (use `runTest` with config instead)
`setStripSizes`	`[sizes...]`	Legacy: Set strip sizes array (use `runTest` with config instead)
`configure`	`{driver, laneSizes, pattern, iterations, shortStripSize?, longStripSize?, testSmallStrips?, testLargeStrips?}`	Set up test parameters (extended with strip size config)
`setLaneSizes`	`[sizes...]`	Set per-lane LED counts directly
`setLedCount`	`count`	Set uniform LED count for all lanes
`setPattern`	`name`	Set test pattern (MSB_LSB_A, SOLID_RGB, etc.)
`setShortStripSize`	`size`	Set short strip LED count
`setLongStripSize`	`size`	Set long strip LED count
`setStripSizeValues`	`short, long`	Set both strip sizes at once
`runParallelTest`	`{drivers: [{driver, laneSizes}, ...], pattern?, timing?}`	Test multiple drivers simultaneously (parallel peripheral coexistence)
`reset`	-	Reset device state

Consolidated `runTest` with Named Arguments (recommended)

json

// Device emits ready event after setup:
RESULT: {"type":"ready","ready":true,"setupTimeMs":2340,"testCases":48,"drivers":3}

// NEW: Single call with config (replaces setDrivers + setLaneRange + setStripSizes + runTest)
// Send:
{"function":"runTest","args":{"drivers":["PARLIO","RMT"],"laneRange":{"min":2,"max":4},"stripSizes":[100,300]}}

// Receive (streaming JSONL):
REMOTE: {"success":true,"streamMode":true}
RESULT: {"type":"test_start","testCases":16}
RESULT: {"type":"test_complete","passed":true,"totalTests":64,"passedTests":64,"durationMs":8230}

Legacy Multi-Call Format (still supported)

json

// Send (4 separate calls):
{"function":"setDrivers","args":["PARLIO"]}
{"function":"setLaneRange","args":[2,4]}
{"function":"setStripSizes","args":[[100,300]]}
{"function":"runTest","args":[]}

// Receive:
REMOTE: {"success":true,"streamMode":true}
RESULT: {"type":"test_start","driver":"PARLIO","totalLeds":100}
RESULT: {"type":"test_complete","passed":true,"totalTests":4,"passedTests":4,"durationMs":2830}

Variable Lane Configurations

python

# Uniform: all lanes same size
config = TestConfig.uniform("RMT", led_count=100, lane_count=4, pattern="MSB_LSB_A")

# Asymmetric: different sizes per lane
config = TestConfig(driver="PARLIO", lane_sizes=[300, 200, 100, 50], pattern="SOLID_RGB")

Strip Size Configuration

python

# Option 1: Individual RPC commands
agent.set_strip_size_values(short=100, long=500)  # Set both sizes at once
agent.set_strip_sizes_enabled(small=True, large=True)  # Enable both strip sizes

# Option 2: Via TestConfig
config = TestConfig(
    driver="PARLIO",
    lane_sizes=[100, 100],
    pattern="MSB_LSB_A",
    short_strip_size=300,      # Override default short strip size
    long_strip_size=1000,      # Override default long strip size
    test_small_strips=True,    # Enable small strip testing
    test_large_strips=True,    # Enable large strip testing (requires sufficient memory)
)
agent.configure(config)

Extending the Protocol

The JSON-RPC architecture allows agents to easily add new test functionality:

Add new RPC handler in examples/Validation/ValidationRemote.cpp
Add corresponding Python method in ci/validation_agent.py
No firmware recompilation needed for client-side extensions

Files:

ci/validation_agent.py - Python ValidationAgent class
ci/validation_loop.py - Test orchestration with CLI
examples/Validation/ValidationRemote.cpp - Firmware RPC handlers

Package Installation Daemon Management

bash daemon <command> - Manage the singleton daemon that handles PlatformIO package installations:

Commands:

bash daemon status - Show daemon status and health
bash daemon stop - Stop the daemon gracefully
bash daemon logs - View daemon log file (last 50 lines)
bash daemon logs-tail - Follow daemon logs in real-time (Ctrl+C to exit)
bash daemon start - Start the daemon (usually automatic)
bash daemon restart - Stop and start the daemon
bash daemon clean - Remove all daemon state (force fresh start)

What is the daemon: The package installation daemon is a singleton background process that ensures PlatformIO package installations complete atomically, even if an agent is interrupted mid-download. It prevents package corruption by:

Surviving agent termination (daemon continues independently)
Preventing concurrent package installations system-wide
Providing progress feedback to waiting clients
Auto-shutting down after 12 hours of inactivity

Note: The daemon starts automatically when needed by bash compile or bash debug. Manual management is typically not required.

Hardware Validation

Hardware Validation

Live Device Testing (AI Agents)

MANDATORY: Driver Selection Required

Usage Examples

Common Options

Strip Size Configuration

Lane Configuration

Color Pattern Configuration

Error Handling

What bash validate Does

Runtime Driver Selection

Why bash validate instead of bash debug

Device Debug (Manual Testing)

JSON-RPC Validation Protocol (Advanced)

Fail-Fast Model

Example JSON-RPC Test Script

EXTENSIBLE TESTING

Python Agent (ci/validation_agent.py)

JSON-RPC Commands

Consolidated runTest with Named Arguments (recommended)

Legacy Multi-Call Format (still supported)

Variable Lane Configurations

Strip Size Configuration

Extending the Protocol

Package Installation Daemon Management

Python Agent (`ci/validation_agent.py`)

Consolidated `runTest` with Named Arguments (recommended)