Back to Ruview

ADR-061: QEMU ESP32-S3 Emulation for Firmware Testing & Development

docs/adr/ADR-061-qemu-esp32s3-firmware-testing.md

0.7.041.2 KB
Original Source

ADR-061: QEMU ESP32-S3 Emulation for Firmware Testing & Development

FieldValue
StatusAccepted
Date2026-03-13 (updated 2026-03-14)
AuthorsRuView Team
RelatesADR-018 (binary frame), ADR-039 (edge intel), ADR-040 (WASM), ADR-057 (build guard), ADR-060 (channel/MAC filter)

Context

The ESP32-S3 CSI node firmware (firmware/esp32-csi-node/) has grown to 16 source files spanning:

ModuleFileTestable in QEMU?
NVS config loadnvs_config.cYes — NVS partition in flash image
Edge processing (DSP)edge_processing.cYes — all math, no HW dependency
ADR-018 frame serializationcsi_collector.c:csi_serialize_frame()Yes — pure buffer ops
UDP stream senderstream_sender.cYes — QEMU has lwIP via SLIRP
WASM runtimewasm_runtime.cYes — CPU only
OTA updateota_update.cPartial — needs HTTP mock
Power managementpower_mgmt.cPartial — no real light-sleep
Display (OLED)display_*.cNo — I2C hardware
WiFi CSI callbackcsi_collector.c:wifi_csi_callback()No — requires RF PHY
Channel hoppingcsi_collector.c:hop_timer_cb()No — requires esp_wifi_set_channel()

Currently, every code change requires flashing to physical hardware on COM7. This creates a bottleneck:

  • Build + flash cycle: ~20 seconds
  • Serial monitor: manual inspection
  • No automated CI (no ESP32-S3 in GitHub Actions runners)
  • Contributors without hardware cannot test firmware changes

Espressif maintains an official QEMU fork (github.com/espressif/qemu) with ESP32-S3 machine support, including dual-core Xtensa LX7, flash mapping, UART, GPIO, timers, and FreeRTOS.

Glossary

TermDefinition
CSIChannel State Information — per-subcarrier amplitude/phase from WiFi
NVSNon-Volatile Storage — ESP-IDF key-value flash partition
TDMTime-Division Multiplexing — nodes transmit in assigned time slots
UARTUniversal Asynchronous Receiver-Transmitter — serial console output
SLIRPUser-mode TCP/IP stack — enables networking without root/TAP
QEMUQuick Emulator — runs ESP32-S3 firmware without physical hardware
QMPQEMU Machine Protocol — JSON-based control interface
LFSRLinear Feedback Shift Register — deterministic pseudo-random generator
SPSCSingle Producer Single Consumer — lock-free ring buffer pattern
FreeRTOSReal-time OS used by ESP-IDF for task scheduling
gcov/lcovGCC code coverage tools for line/branch analysis
libFuzzerLLVM coverage-guided fuzzer for finding crashes
ASANAddressSanitizer — detects buffer overflows and use-after-free
UBSANUndefinedBehaviorSanitizer — detects undefined C behavior

Quick Start

Prerequisites

Install required tools:

bash
# QEMU (Espressif fork with ESP32-S3 support)
git clone https://github.com/espressif/qemu.git
cd qemu && ./configure --target-list=xtensa-softmmu && make -j$(nproc)
export QEMU_PATH=/path/to/qemu/build/qemu-system-xtensa

# ESP-IDF (for building firmware)
# See https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/get-started/

# Python tools
pip install esptool esp-idf-nvs-partition-gen

# Coverage tools (optional, Layer 5)
sudo apt install lcov          # Debian/Ubuntu
brew install lcov              # macOS

# Fuzz testing (optional, Layer 6)
sudo apt install clang         # Debian/Ubuntu

# Mesh testing (optional, Layer 3 — requires root)
sudo apt install socat bridge-utils iproute2

Run the Full Test Suite

bash
# Layer 2: Single-node test (build + run + validate)
bash scripts/qemu-esp32s3-test.sh

# Layer 3: Multi-node mesh (3 nodes, requires root)
sudo bash scripts/qemu-mesh-test.sh 3

# Layer 6: Fuzz testing (60 seconds per target)
cd firmware/esp32-csi-node/test && make all CC=clang
make run_serialize FUZZ_DURATION=60

# Layer 7: Generate NVS test matrix
python3 scripts/generate_nvs_matrix.py --output-dir build/nvs_matrix

# Layer 8: Snapshot regression tests
bash scripts/qemu-snapshot-test.sh --create
bash scripts/qemu-snapshot-test.sh --restore csi-streaming

# Layer 9: Chaos/fault injection
bash scripts/qemu-chaos-test.sh --faults all --duration 120

Environment Variables

VariableDefaultDescription
QEMU_PATHqemu-system-xtensaPath to Espressif QEMU binary
QEMU_TIMEOUT60 (single) / 45 (mesh) / 120 (chaos)Test timeout in seconds
SKIP_BUILDunsetSet to 1 to skip firmware build step
NVS_BINunsetPath to pre-built NVS partition binary
QEMU_NET1Set to 0 to disable SLIRP networking
CHAOS_SEEDcurrent timeSeed for reproducible chaos testing

Exit Codes (all scripts)

CodeMeaningAction
0PASSAll checks passed
1WARNNon-critical issues; review output
2FAILCritical checks failed; fix and re-run
3FATALBuild error, crash, or missing tool; check prerequisites

Decision

Introduce a comprehensive QEMU testing platform for the ESP32-S3 CSI node firmware with nine capability layers:

  1. Mock CSI generator — compile-time synthetic CSI frame injection
  2. QEMU runner — automated build, run, and validation
  3. Multi-node mesh simulation — TDM and aggregation testing across QEMU instances
  4. GDB remote debugging — zero-cost breakpoint debugging without JTAG
  5. Code coverage — gcov/lcov integration for path analysis
  6. Fuzz testing — malformed input resilience for CSI parser, NVS, WASM
  7. NVS provisioning matrix — exhaustive config combination testing
  8. Snapshot & replay — sub-100ms state restore for fast iteration
  9. Chaos testing — fault injection for resilience validation

Layer 1: Mock CSI Generator

Architecture

┌─────────────────────────────────────────────────────┐
│                  ESP32-S3 Firmware                    │
│                                                       │
│  ┌─────────────┐    ┌──────────────────────────────┐ │
│  │  Real WiFi   │    │  Mock CSI Generator          │ │
│  │  CSI Callback │ OR │  (timer → synthetic frames)  │ │
│  │  (HW only)   │    │  (QEMU + unit tests)         │ │
│  └──────┬───────┘    └──────────┬───────────────────┘ │
│         │                       │                     │
│         └───────────┬───────────┘                     │
│                     ▼                                 │
│  ┌──────────────────────────────────────────────────┐ │
│  │  edge_enqueue_csi() → SPSC ring → DSP Core 1    │ │
│  │  ├── Biquad bandpass (breathing / heart rate)    │ │
│  │  ├── Phase unwrapping + Welford stats            │ │
│  │  ├── Top-K subcarrier selection                  │ │
│  │  ├── Presence detection (adaptive threshold)     │ │
│  │  ├── Fall detection (phase acceleration)         │ │
│  │  └── Multi-person vitals clustering              │ │
│  └──────────────────┬───────────────────────────────┘ │
│                     ▼                                 │
│  ┌──────────────────────────────────────────────────┐ │
│  │  csi_serialize_frame() → ADR-018 binary format   │ │
│  │  stream_sender_send() → UDP to aggregator        │ │
│  │  edge vitals packet   → 0xC5110002 (32 bytes)    │ │
│  └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘

Mock CSI Generator Design

When CONFIG_CSI_MOCK_ENABLED=y (Kconfig option), the build replaces esp_wifi_set_csi_config() / esp_wifi_set_csi_rx_cb() with a periodic timer that injects synthetic CSI frames:

c
// mock_csi.c — synthetic CSI frame generator

#define MOCK_CSI_INTERVAL_MS   50   // 20 Hz (matches real CSI rate)
#define MOCK_N_SUBCARRIERS     52   // HT20 mode
#define MOCK_IQ_LEN            (MOCK_N_SUBCARRIERS * 2)  // I + Q bytes

typedef struct {
    uint8_t  scenario;        // 0=empty, 1=person_static, 2=person_walking, 3=fall
    uint32_t frame_count;
    float    person_x;        // Simulated position [0..1]
    float    person_speed;    // Movement speed per frame
    uint8_t  breathing_phase; // Simulated breathing cycle
} mock_state_t;

// Generates realistic CSI I/Q data:
// - Empty room: Gaussian noise + stable phase (low variance)
// - Static person: Phase shift proportional to distance, breathing modulation
// - Walking person: Progressive phase drift + Doppler-like amplitude change
// - Fall event: Sudden phase acceleration spike
void mock_generate_csi_frame(mock_state_t *state, wifi_csi_info_t *out_info);

Signal Model

The synthetic CSI generator models subcarrier amplitude and phase as:

A_k(t) = A_base + A_person * exp(-d_k²/σ²) + noise
φ_k(t) = φ_base + (2π * d / λ) + breathing_mod(t) + noise

where:
  k         = subcarrier index
  d_k       = simulated distance effect on subcarrier k
  A_person  = amplitude perturbation from human body (scenario-dependent)
  d         = simulated person-to-antenna distance
  λ         = wavelength at subcarrier frequency
  breathing_mod(t) = sin(2π * f_breath * t) * amplitude_breath
  noise     = Gaussian, σ tuned to match real ESP32-S3 CSI noise floor (~-90 dBm)

This model exercises:

  • Presence detection (amplitude variance exceeds threshold)
  • Breathing rate extraction (periodic phase modulation at 0.1-0.5 Hz)
  • Fall detection (sudden phase acceleration exceeding fall_thresh)
  • Multi-person separation (distinct subcarrier groups with different breathing frequencies)

Scenarios

IDScenarioDurationExpected Output
0Empty room10spresence=0, motion_energy < thresh
1Static person10spresence=1, breathing_rate ∈ [10,25], fall=0
2Walking person10spresence=1, motion_energy > 0.5, fall=0
3Fall event5sfall=1 flag set, motion_energy spike
4Multi-person15sn_persons=2, independent breathing rates
5Channel sweep5sFrames on channels 1, 6, 11 in sequence
6MAC filter test5sFrames with wrong MAC are dropped (counter check)
7Ring buffer overflow3s1000 frames in 100ms burst, graceful drop
8Boundary RSSI5sRSSI sweeps -90 to -10 dBm, no crash
9Zero-length frame2siq_len=0 frames, serialize returns 0

Layer 2: QEMU Runner & CI

QEMU Runner Script

bash
#!/bin/bash
# scripts/qemu-esp32s3-test.sh

set -euo pipefail

FIRMWARE_DIR="firmware/esp32-csi-node"
BUILD_DIR="$FIRMWARE_DIR/build"
QEMU_BIN="${QEMU_PATH:-qemu-system-xtensa}"
FLASH_IMAGE="$BUILD_DIR/qemu_flash.bin"
LOG_FILE="$BUILD_DIR/qemu_output.log"
TIMEOUT_SEC="${QEMU_TIMEOUT:-60}"

echo "=== QEMU ESP32-S3 Firmware Test ==="

# 1. Build with mock CSI enabled
echo "[1/4] Building firmware (mock CSI mode)..."
idf.py -C "$FIRMWARE_DIR" \
  -D SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.qemu" \
  build

# 2. Merge binaries into single flash image
echo "[2/4] Creating merged flash image..."
esptool.py --chip esp32s3 merge_bin -o "$FLASH_IMAGE" \
  --flash_mode dio --flash_freq 80m --flash_size 8MB \
  0x0     "$BUILD_DIR/bootloader/bootloader.bin" \
  0x8000  "$BUILD_DIR/partition_table/partition-table.bin" \
  0xf000  "$BUILD_DIR/ota_data_initial.bin" \
  0x20000 "$BUILD_DIR/esp32-csi-node.bin"

# 3. Optionally inject pre-provisioned NVS partition
if [ -f "$BUILD_DIR/nvs_test.bin" ]; then
  echo "[2b] Injecting pre-provisioned NVS partition..."
  dd if="$BUILD_DIR/nvs_test.bin" of="$FLASH_IMAGE" \
    bs=1 seek=$((0x9000)) conv=notrunc
fi

# 4. Run in QEMU with timeout, capture UART output
echo "[3/4] Running QEMU (timeout: ${TIMEOUT_SEC}s)..."
timeout "$TIMEOUT_SEC" "$QEMU_BIN" \
  -machine esp32s3 \
  -nographic \
  -drive file="$FLASH_IMAGE",if=mtd,format=raw \
  -serial mon:stdio \
  -no-reboot \
  2>&1 | tee "$LOG_FILE" || true

# 5. Validate expected output
echo "[4/4] Validating output..."
python3 scripts/validate_qemu_output.py "$LOG_FILE"

QEMU sdkconfig overlay (sdkconfig.qemu)

# Enable mock CSI generator (disables real WiFi CSI)
CONFIG_CSI_MOCK_ENABLED=y

# Skip WiFi STA connection (no AP in QEMU)
CONFIG_CSI_MOCK_SKIP_WIFI_CONNECT=y

# Run all scenarios sequentially
CONFIG_CSI_MOCK_SCENARIO=255

# Use loopback for UDP (QEMU SLIRP provides 10.0.2.x network)
CONFIG_CSI_TARGET_IP="10.0.2.2"

# Shorter test durations
CONFIG_CSI_MOCK_SCENARIO_DURATION_MS=5000

# Enable verbose logging for validation
CONFIG_LOG_DEFAULT_LEVEL_INFO=y
CONFIG_CSI_MOCK_LOG_FRAMES=y

Output Validation Script

scripts/validate_qemu_output.py parses the UART log and checks:

CheckPass CriteriaSeverity
Bootapp_main() called, no panic/assertFATAL
NVS loadnvs_config: log line presentFATAL
Mock CSI initmock_csi: Starting mock CSI generatorFATAL
Frame generationmock_csi: Generated N frames where N > 0ERROR
Edge pipelineedge_processing: DSP task started on Core 1ERROR
Vitals outputAt least one vitals: log line with valid BPMERROR
Presence detectionpresence=1 appears during person scenariosWARN
Fall detectionfall=1 appears during fall scenarioWARN
MAC filtercsi_collector: MAC filter dropped N frames where N > 0WARN
ADR-018 serializecsi_collector: Serialized N frames where N > 0ERROR
No crashNo Guru Meditation Error, no assert failed, no abort()FATAL
Clean exitFirmware reaches end of scenario sequenceERROR
Heap OKNo HEAP_ERROR or out of memoryFATAL
Stack OKNo Stack overflow detectedFATAL

Exit codes: 0 = all pass, 1 = WARN only, 2 = ERROR, 3 = FATAL

CI Workflow

yaml
# .github/workflows/firmware-qemu.yml
name: Firmware QEMU Tests
on:
  push:
    paths: ['firmware/**']
  pull_request:
    paths: ['firmware/**']

jobs:
  qemu-test:
    runs-on: ubuntu-latest
    container:
      image: espressif/idf:v5.4
    strategy:
      matrix:
        scenario: [default, nvs-full, nvs-edge-tier0, nvs-tdm-3node]
    steps:
      - uses: actions/checkout@v4

      - name: Install Espressif QEMU
        run: |
          apt-get update && apt-get install -y libslirp-dev libglib2.0-dev ninja-build
          git clone --depth 1 https://github.com/espressif/qemu.git /tmp/qemu
          cd /tmp/qemu
          ./configure --target-list=xtensa-softmmu --enable-slirp
          make -j$(nproc)
          cp build/qemu-system-xtensa /usr/local/bin/
        env:
          QEMU_PATH: /usr/local/bin/qemu-system-xtensa

      - name: Prepare NVS for scenario
        run: |
          case "${{ matrix.scenario }}" in
            nvs-full)
              python firmware/esp32-csi-node/provision.py --dry-run \
                --port dummy --ssid "TestWiFi" --password "test1234" \
                --target-ip "10.0.2.2" --target-port 5005 \
                --channel 6 --filter-mac AA:BB:CC:DD:EE:FF \
                --node-id 1 --edge-tier 2
              cp nvs_provision.bin firmware/esp32-csi-node/build/nvs_test.bin
              ;;
            nvs-edge-tier0)
              python firmware/esp32-csi-node/provision.py --dry-run \
                --port dummy --edge-tier 0 --node-id 5
              cp nvs_provision.bin firmware/esp32-csi-node/build/nvs_test.bin
              ;;
            nvs-tdm-3node)
              python firmware/esp32-csi-node/provision.py --dry-run \
                --port dummy --tdm-slot 1 --tdm-total 3 --node-id 1
              cp nvs_provision.bin firmware/esp32-csi-node/build/nvs_test.bin
              ;;
          esac

      - name: Build firmware (mock CSI mode)
        run: |
          cd firmware/esp32-csi-node
          idf.py -D SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.qemu" set-target esp32s3
          idf.py build

      - name: Run QEMU tests
        run: bash scripts/qemu-esp32s3-test.sh
        env:
          QEMU_PATH: /usr/local/bin/qemu-system-xtensa
          QEMU_TIMEOUT: 90

      - name: Upload QEMU log
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: qemu-output-${{ matrix.scenario }}
          path: firmware/esp32-csi-node/build/qemu_output.log

Layer 3: Multi-Node Mesh Simulation

Run multiple QEMU instances with TAP networking to test TDM slot coordination and multi-node aggregation.

Architecture

┌──────────┐   ┌──────────┐   ┌──────────┐
│ QEMU #0  │   │ QEMU #1  │   │ QEMU #2  │
│ slot=0   │   │ slot=1   │   │ slot=2   │
│ node_id=0│   │ node_id=1│   │ node_id=2│
└────┬─────┘   └────┬─────┘   └────┬─────┘
     │              │              │
     └──────────┬───┴──────────────┘
                ▼
        ┌───────────────┐
        │ TAP bridge    │
        │ (10.0.0.0/24) │
        └───────┬───────┘
                ▼
        ┌───────────────┐
        │ Rust aggregator│
        │ (UDP :5005)   │
        └───────────────┘

Multi-Node Runner

bash
#!/bin/bash
# scripts/qemu-mesh-test.sh — run 3 QEMU nodes + Rust aggregator

set -euo pipefail

N_NODES=${1:-3}
AGGREGATOR_PORT=5005
BRIDGE="qemu-br0"

# Create bridge
ip link add "$BRIDGE" type bridge
ip addr add 10.0.0.1/24 dev "$BRIDGE"
ip link set "$BRIDGE" up

# Build flash images with per-node NVS
for i in $(seq 0 $((N_NODES - 1))); do
  python firmware/esp32-csi-node/provision.py --dry-run \
    --port dummy --node-id "$i" --tdm-slot "$i" --tdm-total "$N_NODES" \
    --target-ip 10.0.0.1 --target-port "$AGGREGATOR_PORT"
  cp nvs_provision.bin "build/nvs_node${i}.bin"

  # Inject NVS into per-node flash image
  cp build/qemu_flash.bin "build/qemu_flash_node${i}.bin"
  dd if="build/nvs_node${i}.bin" of="build/qemu_flash_node${i}.bin" \
    bs=1 seek=$((0x9000)) conv=notrunc
done

# Start Rust aggregator in background
cargo run -p wifi-densepose-hardware --bin aggregator -- \
  --listen 0.0.0.0:${AGGREGATOR_PORT} \
  --expect-nodes "$N_NODES" \
  --output build/mesh_test_results.json &
AGGREGATOR_PID=$!

# Launch QEMU nodes
for i in $(seq 0 $((N_NODES - 1))); do
  TAP="tap${i}"
  ip tuntap add "$TAP" mode tap
  ip link set "$TAP" master "$BRIDGE"
  ip link set "$TAP" up

  qemu-system-xtensa \
    -machine esp32s3 \
    -nographic \
    -drive file="build/qemu_flash_node${i}.bin",if=mtd,format=raw \
    -serial file:"build/qemu_node${i}.log" \
    -nic tap,ifname="$TAP",script=no,downscript=no \
    -no-reboot &
  echo "Started QEMU node $i (PID: $!)"
done

# Wait for test duration
sleep 30

# Validate results
kill $AGGREGATOR_PID 2>/dev/null || true
python3 scripts/validate_mesh_test.py build/mesh_test_results.json --nodes "$N_NODES"

Mesh Validation Checks

CheckPass Criteria
All nodes bootedN distinct node_id values in received frames
TDM orderingSlot 0 frames arrive before slot 1 within each TDM cycle
No slot collisionNo two frames from different nodes with overlapping timestamps within TDM window
Frame count balanceEach node contributes ±10% of total frames
ADR-018 complianceAll frames have valid magic 0xC5110001 and correct node IDs
Vitals per nodeEach node produces independent vitals packets

Layer 4: GDB Remote Debugging

QEMU provides a built-in GDB stub for zero-cost debugging without JTAG hardware.

Usage

bash
# Launch QEMU with GDB stub (paused at boot)
qemu-system-xtensa \
  -machine esp32s3 \
  -nographic \
  -drive file=build/qemu_flash.bin,if=mtd,format=raw \
  -serial mon:stdio \
  -s -S   # -s = GDB on :1234, -S = pause at start

# In another terminal: attach GDB
xtensa-esp-elf-gdb build/esp32-csi-node.elf \
  -ex "target remote :1234" \
  -ex "b edge_processing.c:dsp_task" \
  -ex "b csi_collector.c:wifi_csi_callback" \
  -ex "b mock_csi.c:mock_generate_csi_frame" \
  -ex "watch g_nvs_config.csi_channel" \
  -ex "continue"

Debugging Walkthrough

1. Start QEMU with GDB stub (paused at reset vector):

bash
qemu-system-xtensa \
  -machine esp32s3 \
  -nographic \
  -drive file=build/qemu_flash.bin,if=mtd,format=raw \
  -serial mon:stdio \
  -s -S
# -s  opens GDB server on localhost:1234
# -S  pauses CPU until GDB sends "continue"

2. Connect from a second terminal:

bash
xtensa-esp-elf-gdb build/esp32-csi-node.elf \
  -ex "target remote :1234" \
  -ex "b app_main" \
  -ex "continue"

3. Set a breakpoint on DSP processing and inspect state:

(gdb) b edge_processing.c:dsp_task
(gdb) continue
# ...breakpoint hit...
(gdb) print g_nvs_config
(gdb) print ring->head - ring->tail
(gdb) continue

4. Connect from VS Code using the launch.json config below (set breakpoints in the editor gutter, then press F5).

5. Dump gcov coverage data (requires sdkconfig.coverage overlay):

(gdb) monitor gcov dump
# Writes .gcda files to the build directory.
# Then generate the HTML report on the host:
#   lcov --capture --directory build --output-file coverage.info
#   genhtml coverage.info --output-directory build/coverage_report

Key Breakpoint Locations

BreakpointPurpose
edge_processing.c:dsp_taskDSP consumer loop entry
edge_processing.c:presence_detectThreshold comparison
edge_processing.c:fall_detectPhase acceleration check
csi_collector.c:wifi_csi_callbackFrame ingestion (or mock injection point)
csi_collector.c:csi_serialize_frameADR-018 serialization
nvs_config.c:nvs_config_loadNVS parse logic
wasm_runtime.c:wasm_on_csiWASM module dispatch
mock_csi.c:mock_generate_csi_frameSynthetic frame generation

VS Code Integration

json
// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [{
    "name": "QEMU ESP32-S3 Debug",
    "type": "cppdbg",
    "request": "launch",
    "program": "${workspaceFolder}/firmware/esp32-csi-node/build/esp32-csi-node.elf",
    "miDebuggerPath": "xtensa-esp-elf-gdb",
    "miDebuggerServerAddress": "localhost:1234",
    "setupCommands": [
      { "text": "set remote hardware-breakpoint-limit 2" },
      { "text": "set remote hardware-watchpoint-limit 2" }
    ]
  }]
}

Layer 5: Code Coverage (gcov/lcov)

Build with Coverage

# sdkconfig.coverage (overlay)
CONFIG_COMPILER_OPTIMIZATION_NONE=y
CONFIG_GCOV_ENABLE=y
CONFIG_APPTRACE_GCOV_ENABLE=y

Coverage Collection

bash
# After QEMU run, extract gcov data from flash dump
esptool.py --chip esp32s3 read_flash 0x300000 0x100000 gcov_data.bin

# Or use ESP-IDF's app_trace + gcov integration:
# QEMU + GDB → "monitor gcov dump" → .gcda files

# Generate HTML report
lcov --capture --directory build --output-file coverage.info
lcov --remove coverage.info '*/esp-idf/*' '*/test/*' --output-file coverage_filtered.info
genhtml coverage_filtered.info --output-directory build/coverage_report

Coverage Targets

ModuleTargetCritical Paths
edge_processing.c≥80%dsp_task, biquad_filter, fall_detect, multi_person_cluster
csi_collector.c≥90%csi_serialize_frame, wifi_csi_callback, MAC filter branch
nvs_config.c≥95%Every NVS key read path, default fallback paths
mock_csi.c≥95%All scenarios, all signal model branches
stream_sender.c≥80%Init, send, error paths
wasm_runtime.c≥70%Module load, dispatch, signature verify

Layer 6: Fuzz Testing

Fuzz Targets

TargetInputMutation StrategyLooking For
csi_serialize_frame()Random wifi_csi_info_tExtreme len (0, 65535), NULL buf, negative RSSI, channel 255Buffer overflow, NULL deref
nvs_config_load()Crafted NVS partition binaryTruncated strings, out-of-range u8/u16, missing keys, corrupt headersKconfig fallback, no crash
edge_enqueue_csi()Rapid-fire 10,000 framesVary iq_len (0 to EDGE_MAX_IQ_BYTES+1), randomize RSSIRing overflow, no data corruption
rvf_parser.cMalformed RVF network packetsBad magic, truncated headers, oversized payloadsParse rejection, no crash
wasm_upload.cCorrupt WASM blobsInvalid magic, oversized modules, bad Ed25519 signatures, truncatedRejection without crash, no code execution
csi_serialize_frame() + edge_enqueue_csi()Chained: generate → serialize → enqueueEnd-to-end with random dataPipeline integrity

Implementation Approach

c
// test/fuzz_csi_serialize.c — runs on host (not ESP32)
// Compiled with: clang -fsanitize=fuzzer,address

#include "csi_collector.h"

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    if (size < sizeof(wifi_csi_info_t)) return 0;

    wifi_csi_info_t info;
    memcpy(&info, data, sizeof(info));

    // Point buf at remaining fuzz data
    size_t remaining = size - sizeof(info);
    uint8_t iq_buf[2048];
    if (remaining > sizeof(iq_buf)) remaining = sizeof(iq_buf);
    memcpy(iq_buf, data + sizeof(info), remaining);
    info.buf = iq_buf;
    info.len = (int)remaining;

    uint8_t out[4096];
    csi_serialize_frame(&info, out, sizeof(out));
    return 0;
}

Fuzz CI Job

yaml
  fuzz-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build fuzz targets
        run: |
          cd firmware/esp32-csi-node/test
          clang -fsanitize=fuzzer,address -I../main \
            fuzz_csi_serialize.c ../main/csi_collector.c \
            -o fuzz_serialize
      - name: Run fuzz (5 min per target)
        run: |
          cd firmware/esp32-csi-node/test
          timeout 300 ./fuzz_serialize corpus/ || true
      - name: Upload crashes
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: fuzz-crashes
          path: firmware/esp32-csi-node/test/crash-*

Layer 7: NVS Provisioning Matrix

Config Combinations

ConfigNVS ValuesValidates
default(empty NVS)Kconfig fallback paths
wifi-onlyssid, passwordBasic provisioning
full-adr060channel=6, filter_mac=AA:BB:CC:DD:EE:FFChannel override + MAC filter
edge-tier0edge_tier=0Raw CSI passthrough (no DSP)
edge-tier1edge_tier=1, pres_thresh=100, fall_thresh=2000Stats-only mode
edge-tier2-customedge_tier=2, vital_win=128, vital_int=500, subk_count=16Full vitals with custom params
tdm-3nodetdm_slot=1, tdm_nodes=3, node_id=1TDM mesh timing
wasm-signedwasm_max=4, wasm_verify=1, wasm_pubkey=<32 bytes>WASM with Ed25519 verification
wasm-unsignedwasm_max=2, wasm_verify=0WASM without signature check
5ghz-channelchannel=36, filter_mac=...5 GHz CSI collection
boundary-maxtarget_port=65535, node_id=255, top_k=32, vital_win=256Max-range values
boundary-mintarget_port=1, node_id=0, top_k=1, vital_win=32Min-range values
power-savepower_duty=10, edge_tier=0Low-power mode
corrupt-nvs(manually crafted partial/corrupt partition)Graceful fallback to defaults

Automated Matrix Generation

python
# scripts/generate_nvs_matrix.py
# Generates all 14 NVS partition binaries for CI matrix

CONFIGS = [
    {"name": "default", "args": []},
    {"name": "wifi-only", "args": ["--ssid", "Test", "--password", "test1234"]},
    {"name": "full-adr060", "args": ["--channel", "6", "--filter-mac", "AA:BB:CC:DD:EE:FF",
                                      "--ssid", "Test", "--password", "test"]},
    {"name": "edge-tier0", "args": ["--edge-tier", "0"]},
    # ... all 14 configs
]

Layer 8: Snapshot & Replay

QEMU Snapshot Commands

bash
# Save snapshot after boot + NVS load (skip 3s boot time)
(qemu) savevm post_boot

# Save after WiFi connect + first CSI frame
(qemu) savevm post_connect

# Save after edge pipeline calibration complete (~60s)
(qemu) savevm post_calibration

# Restore any snapshot (< 100ms)
(qemu) loadvm post_connect

Automated Snapshot Pipeline

bash
# scripts/qemu-snapshot-test.sh

# Phase 1: Create base snapshots (one-time, cached in CI)
qemu-system-xtensa ... -monitor unix:qemu.sock,server,nowait &
sleep 5
echo "savevm post_boot" | socat - UNIX-CONNECT:qemu.sock
sleep 10
echo "savevm post_first_frame" | socat - UNIX-CONNECT:qemu.sock

# Phase 2: Run quick tests from snapshots (< 1s each)
for test in test_presence test_fall test_multi_person; do
  echo "loadvm post_first_frame" | socat - UNIX-CONNECT:qemu.sock
  echo "cont" | socat - UNIX-CONNECT:qemu.sock
  sleep 2  # Run test scenario
  # Validate output
done

Performance Impact

OperationWithout SnapshotsWith Snapshots
Full boot + NVS + WiFi mock~5 seconds~5 seconds (first run)
Run single scenario~5s boot + ~5s test = 10s~0.1s restore + ~5s test = 5.1s
Run all 10 scenarios~100 seconds~51 seconds (49% faster)
Run 14 NVS configs × 10 scenarios~23 minutes~12 minutes (48% faster)

Layer 9: Chaos Testing

Fault Injection Table

FaultInjection MethodExpected BehaviorSeverity
WiFi disconnectTimer kills mock WiFi connection after N framesReconnect attempt, CSI pauses and resumesHIGH
Ring buffer overflowBurst 1000 frames in 100msFrame drop counter increments, no crash, no data corruptionHIGH
NVS corruptionFlash image with partial-write NVS partitionFalls back to Kconfig defaults, logs warningMEDIUM
Stack overflowDeep recursion in WASM module callbackWatchdog fires, task restarts, no hangHIGH
Heap exhaustionmalloc returns NULL after N allocationsGraceful degradation, logs OOM, continues operationHIGH
Timer starvationBlock DSP task for 500msFrames dropped from ring, no deadlock, recoversMEDIUM
UDP send failureSLIRP network downstream_sender_send returns -1, error counter incrementsLOW
Corrupt CSI frameInject frame with invalid magic in I/Q dataEdge pipeline rejects, increments error counterLOW
NVS write during readConcurrent NVS open for write while config loadsNo corruption, NVS handle isolationMEDIUM

Chaos Runner

bash
# scripts/qemu-chaos-test.sh

# Run with fault injection enabled
qemu-system-xtensa ... \
  -monitor unix:qemu.sock,server,nowait &

# Inject faults via GDB or monitor commands
for fault in wifi_kill heap_exhaust ring_flood; do
  echo "[CHAOS] Injecting: $fault"
  python3 scripts/inject_fault.py --socket qemu.sock --fault "$fault"
  sleep 5
  python3 scripts/check_health.py --log "$LOG_FILE" --after-fault "$fault"
done

Implementation Plan

PhaseLayerDeliverablesEffortPriority
P1L1 + L2mock_csi.c, mock_csi.h, Kconfig.projbuild, sdkconfig.qemu, qemu-esp32s3-test.sh, validate_qemu_output.py, firmware-qemu.yml2 daysCritical
P2L4 + L5GDB launch config, sdkconfig.coverage, lcov integration, coverage CI job1 dayHigh
P3L7generate_nvs_matrix.py, 14 NVS configs, CI matrix expansion1 dayHigh
P4L6fuzz_csi_serialize.c, fuzz_nvs_config.c, fuzz_edge_enqueue.c, fuzz CI job2 daysHigh
P5L3qemu-mesh-test.sh, TAP bridge setup, validate_mesh_test.py, Rust aggregator integration3 daysHigh
P6L8Snapshot pipeline, cached base images in CI0.5 dayMedium
P7L9inject_fault.py, check_health.py, qemu-chaos-test.sh, 9 fault scenarios2 daysMedium
P8PerformanceInstruction counting, DSP cycle profiling, optimization report1 dayLow

Total: ~12.5 days across 8 phases


File Layout

firmware/esp32-csi-node/
├── main/
│   ├── mock_csi.c              # NEW — synthetic CSI frame generator
│   ├── mock_csi.h              # NEW — mock API + scenario definitions
│   ├── Kconfig.projbuild       # MODIFIED — CONFIG_CSI_MOCK_* options
│   ├── CMakeLists.txt          # MODIFIED — conditional mock_csi.c inclusion
│   └── ... (existing files unchanged)
├── test/
│   ├── fuzz_csi_serialize.c    # NEW — libFuzzer target for serialization
│   ├── fuzz_nvs_config.c       # NEW — libFuzzer target for NVS parsing
│   ├── fuzz_edge_enqueue.c     # NEW — libFuzzer target for ring buffer
│   └── corpus/                 # NEW — seed inputs for fuzz targets
├── sdkconfig.qemu             # NEW — QEMU-specific sdkconfig overlay
├── sdkconfig.coverage         # NEW — gcov-enabled sdkconfig overlay
└── ...

scripts/
├── qemu-esp32s3-test.sh       # NEW — single-node QEMU runner
├── qemu-mesh-test.sh          # NEW — multi-node mesh runner
├── qemu-chaos-test.sh         # NEW — chaos/fault injection runner
├── validate_qemu_output.py    # NEW — UART log validation
├── validate_mesh_test.py      # NEW — mesh test validation
├── generate_nvs_matrix.py     # NEW — NVS config matrix generator
├── inject_fault.py            # NEW — QEMU fault injection
└── check_health.py            # NEW — post-fault health checker

.vscode/
└── launch.json                # MODIFIED — add QEMU GDB debug config

.github/workflows/
└── firmware-qemu.yml          # NEW — CI workflow with matrix

Consequences

Benefits

  1. No hardware required — contributors validate firmware changes with QEMU alone
  2. Automated CI — every PR touching firmware/ runs 14 NVS configs × 10 scenarios in parallel
  3. 10× faster iteration — snapshot restore in <100ms vs 20s flash cycle
  4. Security hardening — fuzz testing catches buffer overflows, NULL derefs, and parser bugs before they reach hardware
  5. Mesh validation — multi-node TDM tested without 3 physical ESP32s
  6. Coverage visibility — lcov reports show untested edge processing paths
  7. Resilience proof — chaos tests verify firmware recovers from WiFi drops, OOM, and ring overflow
  8. GDB debugging — set breakpoints on DSP pipeline without JTAG adapter
  9. Regression detection — boot failures, NVS parsing errors, and FreeRTOS deadlocks caught in CI

Limitations

  1. No real WiFi/CSI — QEMU cannot emulate the ESP32-S3 WiFi radio or CSI extraction hardware
  2. Synthetic CSI fidelity — mock frames approximate real CSI patterns but don't capture real-world multipath, interference, or antenna characteristics
  3. Timing differences — QEMU timing is not cycle-accurate; FreeRTOS tick rates may differ from hardware
  4. No peripheral testing — I2C display, real GPIO, and light-sleep power management cannot be tested
  5. QEMU build requirement — Espressif's QEMU fork must be built from source (not in Ubuntu packages)
  6. Coverage overhead — gcov-enabled builds are ~2× slower in QEMU

What QEMU Testing Covers vs Requires Hardware

Test DomainQEMUHardware
Boot + NVS config (14 configs)FullFull
Edge DSP pipeline (biquad, Welford, top-K)FullFull
ADR-018 frame serializationFullFull
Vitals packet generation (0xC5110002)FullFull
WASM module loading + executionFullFull
Multi-node TDM mesh (3+ nodes)Full (TAP)Full
Fuzz testing (CSI parser, NVS)FullN/A
Code coverage analysisFullPartial
GDB breakpoint debuggingFullFull (JTAG)
Chaos/fault injectionFullManual
OTA update flowPartial (HTTP mock)Full
Real WiFi connectionNoFull
Real CSI data qualityNoFull
Channel hopping on RFNoFull
MAC filter on real framesNoFull
Power management (light-sleep)NoFull
Display rendering (OLED)NoFull
UDP over real networkNoFull

Alternatives Considered

1. Host-native unit tests (no QEMU)

Extract pure C functions (csi_serialize_frame, edge DSP math) and compile/test on host with CMock/Unity. Simpler but doesn't test FreeRTOS integration, NVS, or boot sequence.

Verdict: Complementary — do both. Host unit tests for math, QEMU for integration. Fuzz targets (Layer 6) already use host-native compilation.

2. Hardware-in-the-loop CI (real ESP32 on runner)

Use a self-hosted GitHub Actions runner with a physical ESP32-S3 attached.

Verdict: Valuable but expensive and fragile. QEMU covers ~85% of test cases (up from 70% with all 9 layers). Add HIL later for real CSI validation only.

3. Docker-based ESP-IDF build only (no runtime test)

Just verify the firmware compiles in CI without running it.

Verdict: Already possible but insufficient — compilation doesn't catch runtime bugs (stack overflow, NVS parsing errors, FreeRTOS deadlocks).

4. Renode emulator

Alternative to QEMU with better peripheral modeling for some platforms.

Verdict: Renode has ESP32 support but ESP32-S3 support is less mature than Espressif's own QEMU fork. Revisit if Renode adds full S3 support.


References

  • Espressif QEMU fork — official ESP32/S3/C3/H2 support
  • ESP-IDF QEMU guide
  • libFuzzer documentation — LLVM-based coverage-guided fuzzing
  • lcov — Linux test coverage visualization
  • ADR-018: Binary CSI frame format (magic 0xC5110001)
  • ADR-039: Edge intelligence pipeline (biquad, vitals, fall detection)
  • ADR-040: WASM programmable sensing runtime
  • ADR-057: Build-time CSI guard (CONFIG_ESP_WIFI_CSI_ENABLED)
  • ADR-060: Channel override and MAC address filter

Optimization Log (2026-03-14)

Bugs Fixed

  1. LFSR float biaslfsr_float() used divisor 32767.5 producing range [-1.0, 1.00002]; fixed to 32768.0 for exact [-1.0, +1.0)
  2. MAC filter initializationgen_mac_filter() compared frame_count == scenario_start_ms (count vs timestamp); replaced with boolean flag
  3. Scenario infinite loopadvance_scenario() looped to scenario 0 when all completed; now sets s_all_done=true and timer callback exits early
  4. Boot check severityvalidate_qemu_output.py reported no-boot as ERROR; upgraded to FATAL (nothing works without boot)
  5. NVS boundary configsboundary-max used vital_win=65535 which firmware silently rejects (valid: 32-256); fixed to 256
  6. NVS boundary-minvital_win=1 also invalid; fixed to 32 (firmware min)
  7. edge-tier2-customvital_win=512 exceeded firmware max of 256; fixed to 256
  8. power-save config — Described as "10% duty cycle" but didn't set power_duty=10; fixed
  9. wasm-signed/unsigned — Both configs were identical; signed now includes pubkey blob, unsigned sets wasm_verify=0

Optimizations Applied

  1. SLIRP networking — QEMU runner now passes -nic user,model=open_eth for UDP testing
  2. Scenario completion tracking — Validator now checks All N scenarios complete log marker (check 15)
  3. Frame rate monitoring — Validator extracts scenario=N frames=M counters for rate analysis (check 16)
  4. Watchdog tuningsdkconfig.qemu relaxes WDT to 30s / INT_WDT to 800ms for QEMU timing variance
  5. Timer stack depth — Increased FREERTOS_TIMER_TASK_STACK_DEPTH=4096 to prevent overflow from math-heavy mock callback
  6. Display disabledCONFIG_DISPLAY_ENABLE=n in QEMU overlay (no I2C hardware)
  7. CI fuzz job — Added fuzz-test job running all 3 fuzz targets for 60s each with crash artifact upload
  8. CI NVS validation — Added nvs-matrix-validate job that generates all 14 binaries and verifies sizes
  9. CI matrix expanded — Added edge-tier1, boundary-max, boundary-min to QEMU test matrix (4 → 7 configs)
  10. QEMU cache key — Uses github.run_id with restore-keys fallback to prevent stale QEMU builds