crates/editor/PLAYBACK-FINDINGS.md
SELF-HEALING DOCUMENT: This file is designed to maintain complete context for playback performance work. After any work session, UPDATE this file with your findings before ending.
When your context resets, do this:
PLAYBACK-BENCHMARKS.md for latest raw test data# Check for existing recordings
ls /tmp/cap-real-device-tests/
# If none exist, create them first:
cargo run -p cap-recording --example real-device-test-runner -- baseline --keep-outputs
cargo run -p cap-recording --example playback-test-runner -- full
After completing work, UPDATE these sections:
Last Updated: 2026-01-30
| Metric | Target | MP4 Mode | Fragmented Mode | Status |
|---|---|---|---|---|
| Decoder Init (display) | <200ms | 337ms* | TBD | 🟡 Note |
| Decoder Init (camera) | <200ms | 23ms | TBD | ✅ Pass |
| Decode Latency (p95) | <50ms | 3.1ms | TBD | ✅ Pass |
| Effective FPS | ≥30 fps | 549 fps | TBD | ✅ Pass |
| Decode Jitter | <10ms | ~1ms | TBD | ✅ Pass |
| A/V Sync (mic↔video) | <100ms | 77ms | TBD | ✅ Pass |
| A/V Sync (system↔video) | <100ms | 162ms | TBD | 🟡 Known |
| Camera-Display Drift | <100ms | 0ms | TBD | ✅ Pass |
*Display decoder init time includes multi-position pool initialization (3 decoder instances)
(Update this section as you work)
# Full playback validation (RECOMMENDED)
cargo run -p cap-recording --example playback-test-runner -- full
# Test specific categories
cargo run -p cap-recording --example playback-test-runner -- decoder
cargo run -p cap-recording --example playback-test-runner -- playback
cargo run -p cap-recording --example playback-test-runner -- audio-sync
cargo run -p cap-recording --example playback-test-runner -- camera-sync
# List available recordings
cargo run -p cap-recording --example playback-test-runner -- list
# Test a specific recording
cargo run -p cap-recording --example playback-test-runner -- --recording-path /path/to/recording full
# Save benchmark results to PLAYBACK-BENCHMARKS.md
cargo run -p cap-recording --example playback-test-runner -- full --benchmark-output
# Combined workflow: record then playback
cargo run -p cap-recording --example real-device-test-runner -- baseline --keep-outputs && \
cargo run -p cap-recording --example playback-test-runner -- full
Note: Playback tests require recordings to exist. Run the recording test runner with --keep-outputs first.
| File | Purpose |
|---|---|
crates/rendering/src/decoder.rs | Main decoder interface, spawn_decoder() |
crates/video-decode/src/ | Platform-specific decoders |
crates/video-decode/src/macos.rs | AVAssetReader hardware decoder |
crates/video-decode/src/ffmpeg.rs | FFmpeg software fallback |
crates/audio/src/lib.rs | AudioData loading and sync analysis |
crates/recording/examples/playback-test-runner.rs | Playback benchmark runner |
(Document fixes here as they are implemented)
(Document investigated issues here)
Recording Files (from real-device-test-runner)
├── baseline_mp4/
│ └── content/segments/segment-0/
│ ├── display.mp4 ─┐
│ ├── camera.mp4 ├── Decoder tests
│ ├── audio-input.ogg ─┼── Audio sync tests
│ └── system_audio.ogg ─┘
│
└── baseline_fragmented/
└── content/segments/segment-0/
├── display/ ─┐
│ ├── init.mp4 │ Fragmented decoder
│ └── segment_*.m4s │ (combines init + segments)
├── camera/ ─┘
├── audio-input.m4a ─┬── Audio sync tests
└── system_audio.m4a ─┘
Decoder Pipeline:
┌─────────────────────────────────────────────────────────────────┐
│ spawn_decoder() │
│ ├── macOS: AVAssetReader (VideoToolbox HW accel) │
│ ├── Windows: MediaFoundation (DXVA2/D3D11 HW accel) │
│ └── Fallback: FFmpeg software decoder │
└─────────────────────────────────────────────────────────────────┘
IMPORTANT: Add a new session entry whenever you work on playback performance. This maintains context for future sessions.
### Session YYYY-MM-DD (Brief Description)
**Goal**: What you set out to do
**What was done**:
1. Step 1
2. Step 2
3. ...
**Changes Made**:
- File: description of change
**Results**:
- ✅ What worked
- ❌ What didn't work
**Stopping point**: Where you left off, what to do next
Goal: Establish initial playback performance baseline
What was done:
Changes Made:
crates/editor/PLAYBACK-FINDINGS.md.claude/skills/performance-playback/SKILL.mdResults:
Stopping point: MP4 baseline established. Need to test fragmented mode next.
Goal: Verify current playback performance against targets
What was done:
Changes Made:
Results (Fragmented Mode):
Notes:
Stopping point: All metrics healthy. No action required.
Goal: Verify current playback performance against targets
What was done:
Changes Made:
Results (MP4 Mode):
Analysis:
Stopping point: All metrics healthy. No action required.
Goal: Fix editor playback only achieving ~40-50fps instead of 60fps
What was done:
convert_to_nv12() in frame_ws.rs doing per-pixel CPU color conversion (~6M pixels/frame)Changes Made:
apps/desktop/src-tauri/src/frame_ws.rs: Replaced NV12 conversion with direct RGBA packing in create_watch_frame_ws()apps/desktop/src/utils/socket.ts: Added WebGPU RGBA rendering path using renderFrameWebGPU()Root Cause Analysis: The pipeline was:
The CPU RGBA→NV12 conversion was taking 15-25ms per frame for 3024x1964 resolution, limiting frame rate to 40-50fps. NV12 was originally used to reduce WebSocket bandwidth (12 vs 32 bits/pixel), but the CPU cost outweighed the bandwidth savings for local WebSocket.
Fix: Skip NV12 conversion entirely. Send RGBA directly and use WebGPU renderFrameWebGPU() to display. This trades 2.7x bandwidth increase for eliminating the 15-25ms CPU conversion per frame.
Results:
Stopping point: Fix implemented and compiles. Needs testing with actual editor to verify 60fps achievement.
Goal: Run playback benchmarks, fix panics in decoder fallback path
What was done:
unwrap() on directory paths (fragmented recordings)unwrap() with proper error propagationChanges Made:
crates/video-decode/src/avassetreader.rs: Replaced ffmpeg::format::input(&path).unwrap() and .ok_or(...).unwrap() with map_err()? and ok_or_else()? for clean error propagation instead of panicsResults (MP4 Mode):
Results (Fragmented Mode):
Stopping point: All playback metrics healthy. AVAssetReader panic fixed. No further action needed.
Goal: Comprehensive playback benchmark validation, system audio start_time sync fix
What was done:
unwrap() calls for safetyChanges Made:
crates/recording/src/studio_recording.rs: System audio start_time now syncs to mic (or display) when drift >30ms, matching the existing camera/display sync pattern. Improves playback alignment.Results (MP4 Mode):
Results (Fragmented Mode):
Decoder audit: All unwrap() in avassetreader.rs eliminated. Remaining unwrap() calls in ffmpeg.rs and avassetreader decoder loop are on guaranteed-non-empty BTreeMap caches (safe by construction).
Stopping point: All playback metrics healthy. System audio sync metadata fix applied.
PLAYBACK-BENCHMARKS.md - Raw performance test data (auto-updated by test runner)../recording/FINDINGS.md - Recording performance findings (source of test files)../recording/BENCHMARKS.md - Recording benchmark dataexamples/playback-test-runner.rs - Playback test implementation