docs/adr/ADR-063-mmwave-sensor-fusion.md
Status: Proposed Date: 2026-03-15 Deciders: @ruvnet Related: ADR-014 (SOTA signal processing), ADR-021 (vital sign extraction), ADR-029 (RuvSense multistatic), ADR-039 (edge intelligence), ADR-042 (CHCI coherent sensing)
RuView currently senses the environment using WiFi CSI — a passive technique that analyzes how WiFi signals are disturbed by human presence and movement. While this works through walls and requires no line of sight, CSI-derived vital signs (breathing rate, heart rate) are inherently noisy because they rely on phase extraction from multipath-rich WiFi channels.
A complementary sensing modality exists: 60 GHz mmWave radar modules (e.g., Seeed MR60BHA2) that use active FMCW radar at 60 GHz to measure breathing and heart rate with clinical-grade accuracy. These modules are inexpensive (~$15), run on ESP32-C6/C3, and output structured vital signs over UART.
Live hardware capture (COM4, 2026-03-15) from a Seeed MR60BHA2 on an ESP32-C6 running ESPHome:
[D][sensor:093]: 'Real-time respiratory rate': Sending state 22.00000
[D][sensor:093]: 'Real-time heart rate': Sending state 92.00000 bpm
[D][sensor:093]: 'Distance to detection object': Sending state 0.00000 cm
[D][sensor:093]: 'Target Number': Sending state 0.00000
[D][binary_sensor:036]: 'Person Information': Sending state OFF
[D][sensor:093]: 'Seeed MR60BHA2 Illuminance': Sending state 0.67913 lx
Fusing WiFi CSI with mmWave radar creates a sensor system that is greater than the sum of its parts:
| Capability | WiFi CSI Alone | mmWave Alone | Fused |
|---|---|---|---|
| Through-wall sensing | Yes (5m+) | No (LoS only, ~3m) | Yes — CSI for room-scale, mmWave for precision |
| Heart rate accuracy | ±5-10 BPM | ±1-2 BPM | ±1-2 BPM (mmWave primary, CSI cross-validates) |
| Breathing accuracy | ±2-3 BPM | ±0.5 BPM | ±0.5 BPM |
| Presence detection | Good (adaptive threshold) | Excellent (range-gated) | Excellent + through-wall |
| Multi-person | Via subcarrier clustering | Via range-Doppler bins | Combined spatial + RF resolution |
| Fall detection | Phase acceleration | Range/velocity + micro-Doppler | Dual-confirm reduces false positives to near-zero |
| Pose estimation | Via trained model | Not available | CSI provides pose; mmWave provides ground-truth vitals for training |
| Coverage | Whole room (passive) | ~120° cone, 3m range | Full room + precision zone |
| Cost per node | ~$9 (ESP32-S3) | ~$15 (ESP32-C6 + MR60BHA2) | ~$24 combined |
The RuVector v2.0.4 stack (already integrated per ADR-016) provides the signal processing backbone:
| RuVector Component | Role in mmWave Fusion |
|---|---|
ruvector-attention (bvp.rs) | Blood Volume Pulse estimation — mmWave heart rate can calibrate the WiFi CSI BVP phase extraction |
ruvector-temporal-tensor (breathing.rs) | Breathing rate estimation — mmWave provides ground-truth for adaptive filter tuning |
ruvector-solver (triangulation.rs) | Multilateration — mmWave range-gated distance + CSI amplitude = 3D position |
ruvector-attn-mincut (spectrogram.rs) | Time-frequency decomposition — mmWave Doppler complements CSI phase spectrogram |
ruvector-mincut (metrics.rs, DynamicPersonMatcher) | Multi-person association — mmWave target IDs help disambiguate CSI subcarrier clusters |
The RuvSense multistatic sensing pipeline (ADR-029) gains new capabilities:
| RuvSense Module | mmWave Integration |
|---|---|
pose_tracker.rs (AETHER re-ID) | mmWave distance + velocity as additional re-ID features for Kalman tracker |
longitudinal.rs (Welford stats) | mmWave vitals as reference signal for CSI drift detection |
intention.rs (pre-movement) | mmWave micro-Doppler detects pre-movement 100-200ms earlier than CSI |
adversarial.rs (consistency check) | mmWave provides independent signal to detect CSI spoofing/anomalies |
coherence_gate.rs | mmWave presence as additional gate input — if mmWave says "no person", CSI coherence gate rejects |
The viewpoint fusion pipeline (ruvector/src/viewpoint/) extends naturally:
| Viewpoint Module | mmWave Extension |
|---|---|
attention.rs (CrossViewpointAttention) | mmWave range becomes a new "viewpoint" in the attention mechanism |
geometry.rs (GeometricDiversityIndex) | mmWave cone geometry contributes to Fisher Information / Cramer-Rao bounds |
coherence.rs (phase phasor) | mmWave phase coherence as validation for WiFi phasor coherence |
fusion.rs (MultistaticArray) | mmWave node becomes a member of the multistatic array with its own domain events |
Add 60 GHz mmWave radar sensor support to the RuView firmware and sensing pipeline with auto-detection and device-specific capabilities.
┌─────────────────────────────────────────────────────────┐
│ Sensing Node │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │
│ │ ESP32-S3 │ │ ESP32-C6 │ │ Combined │ │
│ │ WiFi CSI │ │ + MR60BHA2 │ │ S3 + UART │ │
│ │ (COM7) │ │ 60GHz mmWave │ │ mmWave │ │
│ │ │ │ (COM4) │ │ │ │
│ │ Passive │ │ Active radar │ │ Both modes │ │
│ │ Through-wall │ │ LoS, precise │ │ │ │
│ └──────┬───────┘ └──────┬───────┘ └─────┬──────┘ │
│ │ │ │ │
│ └────────┬───────────┘ │ │
│ ▼ │ │
│ ┌────────────────┐ │ │
│ │ Fusion Engine │◄──────────────────────┘ │
│ │ │ │
│ │ • Kalman fuse │ Vitals packet (extended): │
│ │ • Cross-validate│ magic 0xC5110004 │
│ │ • Ground-truth │ + mmwave_hr, mmwave_br │
│ │ calibration │ + mmwave_distance │
│ │ • Fall confirm │ + mmwave_target_count │
│ └────────────────┘ + confidence scores │
└─────────────────────────────────────────────────────────┘
Mode 1: Standalone CSI (existing) — ESP32-S3 only, WiFi CSI sensing.
Mode 2: Standalone mmWave — ESP32-C6 + MR60BHA2, precise vitals in a single room.
Mode 3: Fused (recommended) — ESP32-S3 + mmWave module on UART, or two separate nodes with server-side fusion.
The firmware will auto-detect connected mmWave modules at boot:
0x01 0x01 0x00 0x01 ...) and check for valid response headertypedef struct {
uint8_t mmwave_detected; /** 1 if mmWave module found on UART */
uint8_t mmwave_type; /** Sensor family (MR60BHA2, MR60FDA1, etc.) */
uint8_t mmwave_has_hr; /** Heart rate capability */
uint8_t mmwave_has_br; /** Breathing rate capability */
uint8_t mmwave_has_fall; /** Fall detection capability */
uint8_t mmwave_has_presence; /** Presence detection capability */
uint8_t mmwave_has_distance; /** Range measurement capability */
uint8_t mmwave_has_tracking; /** Multi-target tracking capability */
float mmwave_hr_bpm; /** Latest heart rate from mmWave */
float mmwave_br_bpm; /** Latest breathing rate from mmWave */
float mmwave_distance_cm; /** Distance to nearest target */
uint8_t mmwave_target_count; /** Number of detected targets */
bool mmwave_person_present;/** mmWave presence state */
} mmwave_state_t;
| Sensor | Frequency | Capabilities | UART Protocol | Cost |
|---|---|---|---|---|
| Seeed MR60BHA2 | 60 GHz | HR, BR, presence, illuminance | Seeed proprietary frames | ~$15 |
| Seeed MR60FDA1 | 60 GHz | Fall detection, presence | Seeed proprietary frames | ~$15 |
| Seeed MR24HPC1 | 24 GHz | Presence, sleep stage, distance | Seeed proprietary frames | ~$10 |
| HLK-LD2410 | 24 GHz | Presence, distance (motion + static) | HLK binary protocol | ~$3 |
| HLK-LD2450 | 24 GHz | Multi-target tracking (x,y,speed) | HLK binary protocol | ~$5 |
1. Vital Sign Fusion (Kalman filter)
mmWave HR (high confidence, 1 Hz) ─┐
├─► Kalman fuse → fused HR ± confidence
CSI-derived HR (lower confidence) ─┘
2. Fall Detection (dual-confirm)
CSI phase accel > thresh ──────┐
├─► AND gate → confirmed fall (near-zero false positives)
mmWave range-velocity pattern ─┘
3. Presence Validation
CSI adaptive threshold ────┐
├─► Weighted vote → robust presence
mmWave target count > 0 ──┘
4. Training Calibration
mmWave ground-truth vitals → train CSI BVP extraction model
mmWave distance → calibrate CSI triangulation
mmWave micro-Doppler → label CSI activity patterns
Extend the existing 32-byte vitals packet (magic 0xC5110002) with a new 48-byte fused packet:
typedef struct __attribute__((packed)) {
/* Existing 32-byte vitals fields */
uint32_t magic; /* 0xC5110004 (fused vitals) */
uint8_t node_id;
uint8_t flags; /* Bit0=presence, Bit1=fall, Bit2=motion, Bit3=mmwave_present */
uint16_t breathing_rate; /* Fused BPM * 100 */
uint32_t heartrate; /* Fused BPM * 10000 */
int8_t rssi;
uint8_t n_persons;
uint8_t mmwave_type; /* Sensor type enum */
uint8_t fusion_confidence;/* 0-100 fusion quality score */
float motion_energy;
float presence_score;
uint32_t timestamp_ms;
/* New mmWave fields (16 bytes) */
float mmwave_hr_bpm; /* Raw mmWave heart rate */
float mmwave_br_bpm; /* Raw mmWave breathing rate */
float mmwave_distance; /* Distance to nearest target (cm) */
uint8_t mmwave_targets; /* Target count */
uint8_t mmwave_confidence;/* mmWave signal quality 0-100 */
uint16_t reserved;
} edge_fused_vitals_pkt_t;
_Static_assert(sizeof(edge_fused_vitals_pkt_t) == 48, "fused vitals must be 48 bytes");
New provisioning parameters:
python provision.py --port COM7 \
--mmwave-uart-tx 17 --mmwave-uart-rx 18 \ # UART pins for mmWave module
--mmwave-type auto \ # auto-detect, or: mr60bha2, ld2410, etc.
--fusion-mode kalman \ # kalman, vote, mmwave-primary, csi-primary
--fall-dual-confirm true # require both CSI + mmWave for fall alert
| Phase | Scope | Effort |
|---|---|---|
| Phase 1 | UART driver + MR60BHA2 parser + auto-detection | 2 weeks |
| Phase 2 | Fused vitals packet + Kalman vital sign fusion | 1 week |
| Phase 3 | Dual-confirm fall detection + presence voting | 1 week |
| Phase 4 | HLK-LD2410/LD2450 support + multi-target fusion | 2 weeks |
| Phase 5 | RuVector calibration pipeline (mmWave as ground truth) | 3 weeks |
| Phase 6 | Server-side fusion for separate CSI + mmWave nodes | 2 weeks |