interviews/mobile/02_compute_and_memory.md
This round digs into the constraints that define mobile ML engineering: compute analysis across heterogeneous SoCs, memory management under app lifecycle pressure, numerical precision during format conversion, architecture selection for on-device models, latency budgets under UI thread pressure, and power/thermal trade-offs on battery-powered devices.
➕ Add a Flashcard (Edit in Browser) — see README for the template.
Interviewer: "Your image classification model runs in 4ms on a Pixel 8 Pro (Tensor G3). On a budget phone with a MediaTek Dimensity 700 — which claims 'AI accelerator support' on the spec sheet — the same model takes 40ms. Both phones advertise NPU capability. Why is there a 10× performance gap?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The budget phone's NPU has fewer TOPS." The spec sheets may show similar peak numbers — the issue is deeper than headline TOPS.
Realistic Solution: Five factors create the gap:
(1) Operator coverage — the Tensor G3's NPU supports 95%+ of TFLite operators natively. The Dimensity 700's NPU supports ~60%. Every unsupported op falls back to the CPU, and each fallback incurs a 1-3ms data transfer penalty across the on-chip NoC.
(2) Memory bandwidth — the Pixel 8 Pro has LPDDR5x at 51.2 GB/s. The budget phone has LPDDR4x at 17 GB/s. For memory-bound models, this alone accounts for a 3× difference.
(3) Driver maturity — Google optimizes the Tensor G3's NPU driver for its own TFLite runtime. Third-party SoCs often have less-optimized delegate implementations with higher overhead per inference call.
(4) Thermal throttling — the budget phone has a smaller thermal budget and cheaper cooling. After 10 seconds of continuous inference, it throttles from peak to ~40% of rated performance. The Pixel 8 Pro sustains performance longer with its vapor chamber cooling.
(5) Shared bus contention — the budget phone's NPU shares its memory bus with the camera ISP and display controller. During camera preview, available bandwidth drops further.
Napkin Math: Pixel 8 Pro: 95% NPU delegation, 51.2 GB/s bandwidth, no throttling → 4ms. Budget phone: 60% NPU delegation (40% CPU fallback adds ~15ms), 17 GB/s bandwidth (3× slower memory access adds ~8ms), bus contention adds ~5ms, thermal throttle after 10s adds ~12ms. Total: ~40ms. The "NPU" badge on the spec sheet is marketing, not a performance guarantee.
📖 Deep Dive: Volume I: HW Acceleration
</details>Interviewer: "You're deploying a multi-modal model on a Snapdragon 8 Gen 3 that has a Hexagon NPU (45 TOPS), an Adreno GPU (4.6 TFLOPS FP16), and Kryo CPU cores. The model has Conv2D layers, a custom attention mechanism, GELU activations, and dynamic control flow. Design the execution strategy that minimizes total latency."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Run everything on the NPU — it has the most TOPS." The NPU can't handle all operator types, and forcing unsupported ops onto it causes catastrophic graph partitioning.
Realistic Solution: Partition the model by operator affinity. Note: even when you specify a single compute unit, runtimes like CoreML and TFLite may silently re-partition — CoreML models often execute with "Mixed (Float16, Float32, Int32)" precision even when FP16 is requested, because the Neural Engine lacks native support for certain ops. Always profile with the vendor's tools (Xcode GPU Report, Snapdragon Profiler) to see the actual execution plan.
NPU (Hexagon): Standard Conv2D, depthwise Conv2D, MatMul, ReLU, average/max pooling, concatenation. These are the NPU's sweet spot — fixed-function datapaths optimized for these exact operations. Run the entire convolutional backbone and linear projections here.
GPU (Adreno): Custom attention (Q×K^T softmax, with dynamic shapes), GELU activation (not natively supported on most NPUs), and any ops with non-standard tensor layouts. The GPU is flexible enough to handle these via compute shaders, at moderate power cost.
CPU (Kryo): Dynamic control flow (if/else branching based on intermediate results), non-tensor operations (tokenization, beam search), and pre/post-processing (image resize, NMS, text decoding).
The critical optimization is minimizing partition boundaries. Each NPU→GPU or GPU→CPU handoff costs 0.5-2ms for data transfer across the on-chip NoC. Group all NPU ops contiguously, all GPU ops contiguously. If a single unsupported op sits between two NPU-compatible sections, consider replacing it with a supported approximation (e.g., GELU ≈ x × σ(1.702x) using sigmoid, which the NPU supports) to avoid splitting the graph.
Napkin Math: Model: 50 layers. 40 layers NPU-compatible, 8 GPU (attention + GELU), 2 CPU (control flow). Naive partitioning: 40 NPU layers (8ms) + 2 handoffs (3ms) + 8 GPU layers (6ms) + 1 handoff (1.5ms) + 2 CPU layers (1ms) = 19.5ms. Optimized (replace GELU with sigmoid approximation, fuse attention into fewer GPU calls): 45 NPU layers (9ms) + 1 handoff (1.5ms) + 5 GPU layers (4ms) + 1 handoff (1.5ms) + 2 CPU (1ms) = 17ms. Saving 3 handoffs = 2.5ms = 13% latency reduction.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your team's mobile ML app has a 3-second cold start because it loads 300 MB of model weights into a malloc'd buffer at launch. The PM wants sub-500ms startup. You can't make the model smaller. How do you eliminate the cold start without changing the model?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Load the model in a background thread and show a loading screen." This hides the delay but doesn't eliminate it — the user still waits.
Realistic Solution: Replace malloc + read with mmap(). Memory-mapping the weight file maps it directly into the process's virtual address space without copying any data into physical RAM. The OS loads pages on demand — only the weight pages needed for the currently executing layer are faulted into RAM. First inference is slightly slower (page faults add ~50μs per 4 KB page), but the app is responsive immediately because no upfront loading is needed.
Key benefits: (1) Zero startup cost — the mmap call returns instantly. (2) Graceful under memory pressure — the OS can evict weight pages at any time (they're backed by the file, so no dirty-page write-back). When needed again, they're silently reloaded. Your process is never killed for memory. (3) Shared across processes — if two instances of your model run (e.g., in an app extension), the OS shares the same physical pages.
Trade-off: random access patterns cause excessive page faults. You must ensure the model executes layers sequentially (not randomly accessing distant weights), and the weight file should be stored on fast flash (UFS 4.0: 4.2 GB/s) with minimal fragmentation.
Napkin Math: malloc + read: 300 MB / 2 GB/s (UFS 3.1) = 150ms I/O + 100ms allocation + 200ms framework init = 450ms minimum. With Core ML compilation: +2-5s. mmap: 0ms upfront. First inference: ~200 layers × ~1.5 MB weights per layer × page fault overhead ≈ 50ms extra on first run. Second inference: all pages cached, no overhead. Startup: <50ms vs 3 seconds.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your PM wants to run a 3B parameter LLM on a phone with 8 GB RAM. In FP16, the weights alone are 6 GB. The OS uses 3 GB. You have 5 GB available. Design a memory architecture that makes this work."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "It doesn't fit — tell the PM it's impossible" or "Just quantize to INT4." INT4 (1.5 GB) is one solution, but the PM also wants FP16 quality for premium users. You need a general architecture.
Realistic Solution: Design a paged weight streaming system — the LLM equivalent of virtual memory:
(1) Quantize to INT4 as the default — 3B × 0.5 bytes = 1.5 GB. Fits entirely in RAM with 3.5 GB headroom. This serves 90% of users.
(2) For FP16 quality, stream weights from flash — partition the 6 GB of FP16 weights into 256 MB chunks (one chunk per ~4 transformer blocks). Allocate a 1 GB weight buffer in RAM. At any time, only the currently-executing blocks' weights are resident. As the model advances through layers, prefetch the next chunk from flash while the current chunk executes. UFS 4.0 reads at 4.2 GB/s → 256 MB loads in 61ms. If one transformer block takes ~15ms to execute and you have 4 blocks per chunk (60ms compute), the prefetch completes before the next chunk is needed — zero stall.
(3) KV-cache budget — at 2048 context length: 3B model with 32 layers, 32 heads, 128 dim per head. KV-cache = 2 × 32 × 32 × 128 × 2048 × 2 bytes = 1.07 GB in FP16, or 268 MB in INT8 (quantized KV-cache). Use INT8 KV-cache to fit within budget.
(4) Total memory — INT4 path: 1.5 GB weights + 268 MB KV-cache + 100 MB activations + 50 MB runtime = 1.92 GB. FP16 streaming path: 1 GB weight buffer + 268 MB KV-cache + 100 MB activations + 50 MB runtime = 1.42 GB resident (6 GB on flash).
Napkin Math: FP16 weights: 3B × 2 = 6 GB (doesn't fit in 5 GB available). INT4 weights: 3B × 0.5 = 1.5 GB ✓. FP16 streaming: 1 GB buffer, 256 MB chunks, UFS 4.0 at 4.2 GB/s → 61ms per chunk load. 4 blocks × 15ms = 60ms compute per chunk. Prefetch hides latency: 61ms load overlaps with 60ms compute → ~1ms stall per chunk. 32 layers / 4 per chunk = 8 chunks per forward pass. Total stall: ~8ms per token. Token latency: 60ms compute + 8ms stall = 68ms/token (vs 60ms if all weights were resident). Acceptable.
📖 Deep Dive: Volume II: Edge Intelligence
</details>Interviewer: "You convert a PyTorch model from FP32 to Core ML FP16 for the Apple Neural Engine. Overall accuracy drops 0.3% — acceptable. But one specific layer's output diverges by 12% from the FP32 reference. Which layer type is most likely the culprit, and why?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The convolution layers are losing precision." Standard Conv2D layers are highly robust to FP16 — their outputs are sums of many small products, which average out rounding errors.
Realistic Solution: The most likely culprits are BatchNorm and activation functions with wide dynamic range. BatchNorm computes $\hat{x} = (x - \mu) / \sqrt{\sigma^2 + \epsilon}$. In CoreML's FP16 conversion, a documented bug causes the epsilon parameter to remain FP32 while mean values are cast to FP16, creating type mismatches that produce large errors (Apple coremltools issues #2470, #2625). Even without the bug, dividing by a small variance in FP16 amplifies rounding errors.
Activation functions are another real-world trap: Mish (x × tanh(softplus(x))) and hard-swish in MobileNetV3 produce mean absolute errors exceeding 1.0 in intermediate layers when run in FP16 on the Neural Engine (coremltools issue #2359). The chained nonlinearities (exp, tanh, multiply) compound FP16 rounding at each step.
Other culprits: (1) Softmax — exp() amplifies small input differences. (2) Large logits — values exceeding FP16 max (65504) overflow to infinity. (3) Residual connections — adding a large tensor to a small one causes catastrophic cancellation.
Fix: use mixed precision — keep BatchNorm, problematic activations (Mish, hard-swish), softmax, and the final projection in FP32 while running everything else in FP16. Core ML supports per-layer precision specification via compute_units and typed execution.
Napkin Math: LayerNorm with σ² = 1e-4. FP32 precision: ~7 decimal digits → division accurate to 0.0001%. FP16 precision: ~3 decimal digits → division accurate to 0.1%. Relative error amplification: 0.1% / 0.0001% = 1000×. If the layer output range is [0, 1], a 0.1% error = 0.001 absolute. After 10 subsequent layers each amplifying by 1.1×: 0.001 × 1.1¹⁰ = 0.0026 → 12% divergence on a feature with range [0, 0.02] is entirely plausible.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "You're deploying a 100-layer model on the Apple Neural Engine via Core ML. Running everything in FP16 gives a 2% accuracy drop — unacceptable for your medical imaging app. Running everything in FP32 means the Neural Engine can't be used (it only supports FP16). Design a mixed-precision strategy that gets Neural Engine speed with FP32 accuracy."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Run the whole model in FP32 on the GPU." This works but is 3-5× slower than the Neural Engine. You're leaving performance on the table.
Realistic Solution: Profile each layer's precision sensitivity by running the model in FP32 and comparing each layer's output to its FP16 equivalent using cosine similarity. Layers with cosine similarity > 0.999 are FP16-safe. Layers below that threshold need FP32.
Typical sensitivity profile: FP16-safe (95% of layers): Conv2D, depthwise Conv2D, ReLU, average pooling, concatenation, linear projections. These layers' outputs are sums of many products — rounding errors average out. FP32-required (5% of layers): LayerNorm (division by small variance), softmax (exponential amplification), the final classification head (small differences in logits change the predicted class), and any layer immediately after a residual addition with large magnitude difference.
Execution plan: the FP16-safe layers run on the Neural Engine. At each FP32-required layer, data transfers to the CPU/GPU for FP32 computation, then returns to the Neural Engine. Each transfer costs ~1-2ms. With 5 FP32 layers: 5 × 2 round-trips × 1.5ms = 15ms overhead. Total: 20ms (Neural Engine) + 15ms (transfers) + 3ms (FP32 compute) = 38ms. Compare to: all-FP16 Neural Engine = 20ms (but 2% accuracy loss), all-FP32 GPU = 100ms. The mixed approach gives 98% of FP32 accuracy at 38% of FP32 latency.
Napkin Math: 100 layers. 95 on Neural Engine FP16: 95 × 0.2ms = 19ms. 5 on CPU FP32: 5 × 0.6ms = 3ms. 10 data transfers (5 round-trips): 10 × 1.5ms = 15ms. Total: 37ms. All-FP16: 20ms (fast but inaccurate). All-FP32 GPU: 100ms (accurate but slow). Mixed: 37ms — 1.85× slower than all-FP16, but 2.7× faster than all-FP32, with full accuracy.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "MobileNetV3 adds squeeze-and-excitation (SE) blocks that increase FLOPs by 2%. Your colleague says 'that's wasted compute — remove them to speed up inference.' Why is your colleague wrong?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "2% more FLOPs = 2% slower inference." This treats all FLOPs as equal, ignoring the accuracy-per-FLOP trade-off.
Realistic Solution: SE blocks are a channel attention mechanism: global average pool → small FC layer → ReLU → small FC layer → sigmoid → channel-wise multiply. They learn which channels matter for each input, effectively giving the network input-dependent feature selection. The 2% FLOP increase buys a 2-3% accuracy improvement. This means you can use a smaller base model (e.g., MobileNetV3-Small instead of MobileNetV2-1.0) and still hit the same accuracy target — saving 30%+ FLOPs overall. The SE block's operations (global pool, small FC) are also extremely NPU-friendly — they map to a few MAC operations on the Neural Engine with near-zero overhead.
The deeper insight: on mobile, the goal is maximum accuracy per milliwatt, not minimum FLOPs. A 2% FLOP increase that enables a 30% smaller base model is a massive win for battery life.
Napkin Math: MobileNetV2-1.0: 300 MFLOPs, 72% top-1 ImageNet. MobileNetV3-Small with SE: 56 MFLOPs, 67.4% top-1. MobileNetV3-Large with SE: 219 MFLOPs, 75.2% top-1. To match MobileNetV2's 72% accuracy: MobileNetV3 needs ~150 MFLOPs (with SE) vs 300 MFLOPs (without SE). The 2% FLOP overhead of SE enables a 50% total FLOP reduction at equal accuracy.
📖 Deep Dive: Volume I: Network Architectures
</details>Interviewer: "Your PM saw Apple's on-device LLM demo and wants you to ship a 3B parameter chatbot in your app by next quarter. Walk through the feasibility analysis — memory, compute, latency, and battery — and tell the PM what's actually possible."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "3B parameters is too big for a phone — it's impossible." It's not impossible, but the constraints are severe and the PM needs to understand the trade-offs.
Realistic Solution: Walk through each constraint:
Memory: 3B params in FP16 = 6 GB. iPhone 15 Pro has 8 GB RAM, ~5 GB available. Doesn't fit. INT4 quantization: 3B × 0.5 bytes = 1.5 GB weights. Plus KV-cache for 2048 context: ~270 MB (INT8). Plus activations: ~100 MB. Plus runtime: ~50 MB. Total: 1.92 GB. Fits with 3 GB headroom.
Compute: Autoregressive decoding: ~2 × 3B = 6 GFLOPs per token. Apple A17 Pro Neural Engine: ~35 TOPS. But LLM decoding is memory-bandwidth bound (loading all weights per token). At INT4: 1.5 GB weights / 77 GB/s (LPDDR5x) = 19.5ms per token = ~51 tokens/second. Feels responsive.
Latency: Prefill (processing the prompt): 512 input tokens × 6 GFLOPs = 3.07 TFLOPs. At 35 TOPS: ~88ms. Acceptable. Decode: 19.5ms/token. 100-token response: ~2 seconds. Acceptable.
Battery: INT4 inference at ~3W (NPU). 100-token response: 2 seconds × 3W = 6 joules. iPhone 15 Pro battery: 17.3 Wh = 62,280 J. Each response costs 6/62,280 = 0.01% battery. 100 conversations/day = 1% battery. Acceptable.
Verdict: Feasible with INT4 quantization. Quality will be noticeably worse than cloud GPT-4, but usable for simple tasks. Ship it as a "fast local mode" with cloud fallback for complex queries.
Napkin Math: Memory: 1.92 GB (INT4) ✓. Decode: 19.5ms/token → 51 tok/s ✓. Prefill: 88ms for 512 tokens ✓. Battery: 1% per 100 conversations ✓. App size: 1.5 GB model (download on WiFi, not in app bundle). Feasible with caveats.
📖 Deep Dive: Volume II: Edge Intelligence
</details>Interviewer: "You're building live captions for a video calling app. Your team is debating between Whisper-small (244M params, non-streaming) and a custom RNN-T model (30M params, streaming). The PM wants 'the best accuracy.' Why might the smaller model be the right choice?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Whisper is more accurate on benchmarks, so use Whisper." Benchmark accuracy doesn't account for the system-level constraints of real-time captioning.
Realistic Solution: Whisper is a sequence-to-sequence model that processes audio in 30-second chunks. For live captions, this means: (1) 30-second latency — the user speaks, and captions appear 30 seconds later. Unusable for live conversation. (2) Memory: 244M params × 2 bytes (FP16) = 488 MB always resident, plus attention KV-cache for 30 seconds of audio. (3) Compute burst: processing 30 seconds of audio at once requires a large compute burst, causing thermal spikes.
The RNN-T model processes 80ms audio frames incrementally: (1) 200ms latency — captions appear within 200ms of speech, feeling real-time. (2) Memory: 30M × 2 = 60 MB weights, plus minimal hidden state (~1 MB). (3) Steady compute: small, constant inference every 80ms — no thermal spikes, predictable power draw.
Accuracy comparison: Whisper-small WER ~8% on LibriSpeech. RNN-T WER ~12%. But for live captions, the 4% WER gap is invisible to users because: (a) captions are read in real-time where context helps comprehension, (b) 30-second delayed captions are functionally useless regardless of accuracy. The streaming model wins on the metric that matters: usable accuracy at acceptable latency.
Napkin Math: Whisper: 488 MB memory, 30s latency, 8% WER. Burst power: ~5W for 3 seconds every 30 seconds. RNN-T: 60 MB memory (8× less), 200ms latency (150× less), 12% WER. Steady power: ~0.5W continuous. Battery for 1-hour call: Whisper ~5W × 0.1 duty = 0.5W avg → 0.5 Wh. RNN-T: 0.5W × 1.0 duty = 0.5W avg → 0.5 Wh. Similar battery, but RNN-T frees 428 MB of RAM for other apps.
📖 Deep Dive: Volume I: Network Architectures
</details>Interviewer: "Design an inference pipeline for a camera app that maintains 60 FPS preview while running a 50ms portrait segmentation model. The model cannot run every frame. How do you architect this so the user sees smooth video with accurate segmentation?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Run the model every 3rd frame and hold the mask for the other 2 frames." This works but produces visible mask "popping" — the segmentation boundary jumps every 3 frames instead of moving smoothly.
Realistic Solution: Design a triple-buffer async pipeline with temporal interpolation:
Buffer A (Display): The frame currently being shown to the user. Always available, never blocked.
Buffer B (Processing): The frame currently being processed by the ML model on the NPU. Takes 50ms.
Buffer C (Queued): The most recent camera frame, waiting to be processed when the NPU finishes Buffer B.
The camera produces frames at 60 FPS (every 16.67ms). Every frame goes to Buffer C (overwriting the previous queued frame). When the NPU finishes Buffer B, it immediately starts on Buffer C. The display thread composites Buffer A's camera frame with the most recent completed segmentation mask.
Temporal interpolation: Between mask updates (every ~50ms = every 3rd frame), use optical flow or simple affine transform to warp the previous mask to match the current frame's motion. This makes the mask boundary move smoothly at 60 FPS even though the model only runs at 20 FPS. Cost: ~2ms per frame for the warp on the GPU.
Result: 60 FPS smooth preview, 20 FPS mask updates, smooth mask boundaries via interpolation, zero jank. The user perceives real-time segmentation.
Napkin Math: Camera: 60 FPS → frame every 16.67ms. Model: 50ms → 20 FPS. Mask interpolation: 2ms per frame on GPU. Display thread: 7ms (UI) + 2ms (interpolation) + 1ms (composite) = 10ms < 16.67ms budget ✓. NPU utilization: 50ms/50ms = 100% (always processing). Perceived quality: smooth 60 FPS with mask updating every 50ms (3 frames). Without interpolation: mask "jumps" every 3 frames. With interpolation: mask moves smoothly every frame.
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "Your ML-powered fitness app drains 1% battery per minute during a workout. The PM says 'the model is too expensive — optimize it.' You profile the power breakdown and find the model is not the main culprit. What is?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The ML model must be the power hog — it's the most computationally intensive component." This assumes compute = power, ignoring the rest of the system.
Realistic Solution: Profile the full system power breakdown during a workout session: (1) GPS radio: continuously active for route tracking — ~200 mW. (2) Screen: always on showing the workout dashboard at high brightness (outdoor use) — ~800 mW. (3) Heart rate sensor: continuous optical sensing — ~100 mW. (4) Cellular/WiFi radio: uploading telemetry every 5 seconds — ~500 mW average. (5) ML inference: pose estimation model running every 500ms at 5ms per inference — duty cycle of 1%. NPU power during inference: ~2W. Average: 2W × 0.01 = 20 mW.
Total: 200 + 800 + 100 + 500 + 20 = 1620 mW. The ML model accounts for 20/1620 = 1.2% of total power. Optimizing the model to zero would save 1.2% of battery drain. The real levers: dim the screen (save 400 mW), reduce GPS polling rate (save 100 mW), batch telemetry uploads (save 300 mW).
Napkin Math: iPhone 15 battery: 3349 mAh × 3.83V = 12.8 Wh. At 1620 mW total draw: 12.8 Wh / 1.62W = 7.9 hours. 1% per minute = 100 minutes to drain → 1.62W × (100/60) = 2.7W total (our estimate is conservative). Even doubling model efficiency saves: 20 mW / 2700 mW = 0.7% of battery life. Dimming the screen saves 15%.
📖 Deep Dive: Volume II: Sustainable AI
</details>Interviewer: "Your model runs at 2 TOPS on both the Snapdragon 8 Gen 3's Hexagon NPU and its Adreno GPU. Same model, same throughput. But the NPU uses 0.5W while the GPU uses 1.5W. Why does the NPU achieve 3× better TOPS/W for the same workload?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The NPU has a newer manufacturing process." Both are on the same 4nm die — the difference is architectural.
Realistic Solution: The NPU and GPU achieve the same TOPS but through fundamentally different architectures:
GPU (Adreno): A general-purpose SIMD processor. Every MAC operation requires: (1) instruction fetch and decode, (2) register file read (source operands), (3) ALU execution, (4) register file write (result), (5) thread scheduling and warp management. The control logic (instruction decoder, scheduler, branch predictor) consumes power even though it does no useful math. For a simple multiply-accumulate, ~60% of energy goes to data movement and control, ~40% to the actual computation.
NPU (Hexagon): A fixed-function accelerator with hardwired datapaths for MAC operations. There's no instruction fetch per operation — the dataflow is configured once at model load time. Data moves through a spatial pipeline of MAC units with minimal control overhead. ~85% of energy goes to computation, ~15% to data movement. No branch prediction, no thread scheduling, no instruction cache.
The result: for the specific operations the NPU supports (Conv2D, MatMul, pooling), it achieves 3-5× better energy efficiency than the GPU. The GPU's flexibility is its strength for general compute but its weakness for the narrow, repetitive operations that dominate neural network inference.
Napkin Math: GPU at 2 TOPS, 1.5W: 1.33 TOPS/W. Energy per MAC: 1.5W / (2 × 10¹² ops/s) = 0.75 pJ/op. NPU at 2 TOPS, 0.5W: 4.0 TOPS/W. Energy per MAC: 0.5W / (2 × 10¹²) = 0.25 pJ/op. The NPU is 3× more efficient per operation. Over a 1-hour session at continuous inference: GPU = 1.5 Wh, NPU = 0.5 Wh. On a 12.8 Wh battery: GPU uses 11.7% battery, NPU uses 3.9%.
Key Equation: $\text{Energy per op} = \frac{P_{\text{total}}}{\text{TOPS} \times 10^{12}}\ \text{(joules/op)}$
📖 Deep Dive: Volume I: HW Acceleration
</details>Interviewer: "You convert your PyTorch model to TFLite and enable the Qualcomm QNN delegate for the Hexagon NPU. The delegate reports '87% of ops delegated.' Your colleague says 'close enough — the remaining 13% will run on CPU, no big deal.' Why is 87% delegation potentially worse than 0% delegation?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "87% delegation means 87% of compute runs on the NPU." Delegation percentage counts ops, not compute time. And partial delegation creates expensive partition boundaries.
Realistic Solution: The 13% undelegated ops are scattered throughout the graph, not clustered at the end. Each undelegated op creates a partition boundary: NPU→CPU→NPU. If there are 8 undelegated ops in the middle of the graph, you get 9 NPU subgraphs with 8 round-trip data transfers. Each transfer costs 1-3ms. Total transfer overhead: 8 × 2ms = 16ms — potentially more than the entire model would take on CPU alone.
Other failure categories: (1) Op variant mismatch — NPU supports Conv2D but not Conv2D with dilation > 1. (2) Shape constraints — NPU requires dimensions to be multiples of 4 or 8. (3) Quantization mismatch — NPU expects per-channel symmetric INT8 but model uses per-tensor asymmetric. (4) Layout incompatibility — NPU operates in NHWC but model expects NCHW.
The fix: use vendor profiling tools (Snapdragon Profiler, Xcode Instruments) to inspect the actual execution plan. Target 100% delegation or cluster all undelegated ops at the end.
Napkin Math: Full NPU delegation: 4ms. 87% delegation with 8 partition boundaries: 9 NPU segments (3.5ms) + 8 CPU ops (0.8ms) + 8 transfers (16ms) + lost optimizations (2ms) = 22.3ms. Full CPU fallback: 15ms. Partial delegation is 1.5× slower than no delegation at all.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "You're running a 3B parameter LLM (Phi-3-mini, INT4) on an iPhone 15 Pro with 8 GB RAM. The model weights take 1.5 GB. Short conversations work fine, but after 10+ back-and-forth turns, the app gets killed by iOS. The model weights haven't changed. What's growing?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "There's a memory leak in the inference engine." The memory growth is by design, not a bug.
Realistic Solution: The KV-cache. During autoregressive generation, the model stores key and value tensors for every token in the conversation history. For Phi-3-mini (32 layers, 32 heads, head_dim=96, FP16 KV): KV-cache per token = 2 × 32 × 32 × 96 × 2 bytes = 393 KB. After 10 turns (~2000 tokens): 2000 × 393 KB = 786 MB. Add model weights (1.5 GB), iOS overhead (~3 GB), and app runtime (~200 MB): total = 5.5 GB. iOS jetsam threshold is ~6 GB for foreground apps. By turn 15 (~3000 tokens): KV-cache = 1.15 GB, total = 5.85 GB — jetsam kills the app.
Fixes: (1) Quantize KV-cache to INT8 — halves KV memory. (2) Sliding window attention — keep only the last 1024 tokens. (3) Grouped-Query Attention (GQA) — Phi-3 uses 8 KV heads instead of 32, reducing KV-cache by 4×: 786 MB → 196 MB. (4) Hard context limit with auto-summarization.
Napkin Math: Phi-3-mini with GQA (8 KV heads): KV per token = 2 × 32 × 8 × 96 × 2 = 98 KB. At 2048 tokens: 196 MB. With INT8 KV: 98 MB. Total: 1.5 GB + 98 MB + 200 MB + 3 GB = 4.8 GB. Safe on 8 GB device.
Key Equation: $\text{KV-cache} = 2 \times L \times H_{kv} \times d_h \times n_{\text{tokens}} \times \text{bytes}$
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "You are running inference for an NLP model entirely on the CPU of a mobile Snapdragon SoC, which features an octa-core big.LITTLE architecture (1 Prime core, 3 Performance cores, 4 Efficiency cores). You spawn 8 threads to maximize throughput using OpenMP. Strangely, using exactly 4 threads is significantly faster than using all 8 threads. Why?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Assuming all CPU cores are identical and that more threads always equal more performance, like on a standard desktop x86 chip."
Realistic Solution: You fell into the heterogeneous synchronization trap. Big.LITTLE architectures mix massive, power-hungry cores (Prime/Performance) with tiny, slow cores (Efficiency). When you split a matrix multiplication across 8 threads, the framework divides the work into 8 equal chunks. The Prime and Performance cores finish their chunks in a few milliseconds. However, they must then wait at a synchronization barrier for the 4 Efficiency cores, which are physically 3x to 4x slower, to finish their chunks. The fast cores sit completely idle, blocked by the slowest cores on the chip.
Napkin Math: Let's say a layer takes 8000 operations. With 4 fast cores, each does 2000 ops at 1000 ops/ms ->
Latency = 2ms. If you use 8 cores (4 fast, 4 slow), they each get 1000 ops. The fast cores finish in1ms. The slow cores (running at 200 ops/ms) take5ms. Because of the sync barrier, the layer takes5ms. By adding more cores, you made the system 2.5x slower.
📖 Deep Dive: Volume I: HW Acceleration
</details>Interviewer: "You deploy an INT8 quantized image classification model to a mobile phone. During local testing with PyTorch, the model has 95% accuracy. On the phone, using TFLite, it drops to 82%. You discover that a specific intermediate activation tensor is consistently hitting exactly 0 when it shouldn't. What quantization hardware mismatch occurred?"
Common Mistake: "Assuming INT8 is a universal format and that 'quantized' means the same thing across all frameworks and hardware."
Realistic Solution: You hit an asymmetric vs. symmetric quantization mismatch, specifically regarding the zero-point. In PyTorch, you likely simulated symmetric quantization, where 0 in floating-point maps exactly to 0 in INT8. However, TFLite on mobile often uses asymmetric quantization, where the minimum and maximum float values are mapped to -128 and 127, and the floating-point 0.0 might map to an integer like 14. If the NPU hardware or driver assumes a symmetric format, it will not apply the zero-point offset during the matrix multiplication. All negative activations (like those before a ReLU) are incorrectly clipped, physically destroying the feature map.
Napkin Math: $Quantized = ext{round}(Float / Scale) + ZeroPoint$. If $Scale = 0.1$ and $ZeroPoint = 50$, a float of $0.0$ should be $50$. If the hardware hardware ignores the zero point (treating it as $0$), it computes $-50$ and clips it to
0. The entire dynamic range of your activation is shifted and destroyed at the hardware level.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "Your team is shipping an on-device LLM feature on the iPhone 16 Pro, which has 8 GB LPDDR5X at 51.2 GB/s peak bandwidth. The model is a 4B parameter LLM quantized to INT4 (2 GB weights). During autoregressive decoding, every token requires loading all weights once. Is the memory bandwidth sufficient for real-time token generation at 30+ tokens/second? Show your math."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "51.2 GB/s is plenty — 2 GB of weights at 51.2 GB/s means each token takes 39ms, so ~25 tokens/second. Close enough." This uses peak bandwidth, which is never achievable in practice. Real sustained bandwidth is 60-70% of peak due to refresh cycles, bank conflicts, and bus contention from the OS, display controller, and ISP sharing the same memory bus.
Realistic Solution: Autoregressive LLM decoding is almost entirely memory-bandwidth bound — each token requires reading all model weights once (the compute-to-byte ratio is ~1 op/byte for INT4 GEMV). The arithmetic is fast; the bottleneck is feeding data to the compute units.
Sustained bandwidth on iPhone 16 Pro: ~32-35 GB/s (65-70% of peak). The OS, display pipeline, and background tasks consume ~5 GB/s, leaving ~27-30 GB/s for your model. At 2 GB per token load: 2 GB / 28 GB/s ≈ 71ms per token ≈ 14 tokens/second. This is below the 30 tok/s target.
Optimizations: (1) Group quantization with INT4 + INT8 KV-cache — reduces effective weight reads via block-sparse patterns. (2) Speculative decoding — draft model (200 MB) generates 4 candidate tokens, verified in a single forward pass of the large model. Amortizes weight loading over multiple tokens: effective rate = 14 × 4 = ~40 tokens/second after verification rejection. (3) Weight prefetching — overlap weight loading for layer N+1 with computation of layer N, hiding ~30% of memory latency. (4) Reduce model to 3B params — 1.5 GB weights → 1.5 / 28 = 54ms → 18.5 tok/s base, ~55 tok/s with speculative decoding.
Napkin Math: Peak BW: 51.2 GB/s. Sustained: ~28 GB/s (after OS/display contention). Weight load per token: 2 GB (INT4, 4B params). Time per token: 2 / 28 = 71ms → 14 tok/s. With speculative decoding (4× acceptance): 14 × 4 = ~40 tok/s (after ~30% rejection). With 3B model: 1.5 / 28 = 54ms → 18.5 tok/s base → ~55 tok/s speculative. The bandwidth wall, not compute, determines on-device LLM speed.
📖 Deep Dive: Volume I: HW Acceleration
</details>Interviewer: "Apple's A17 Pro has a Neural Engine rated at 35 TOPS (INT8) drawing ~2W, and an Apple GPU at ~2.15 TFLOPS (FP16) drawing ~5W. A colleague says 'always use the ANE — it's 8× more efficient per watt.' Calculate the TOPS/W for each, and describe a realistic scenario where the GPU wins despite lower power efficiency."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The ANE is always better because 35 TOPS / 2W = 17.5 TOPS/W beats the GPU's 2.15 TFLOPS / 5W = 0.43 TFLOPS/W." This compares INT8 TOPS to FP16 TFLOPS — apples to oranges — and ignores that TOPS/W only matters when the hardware can actually execute your workload.
Realistic Solution: First, normalize the comparison. ANE: 35 TOPS (INT8) at 2W → 17.5 TOPS/W. GPU: 2.15 TFLOPS (FP16) at 5W → 0.43 TFLOPS/W. In comparable INT8 terms, the GPU achieves ~4.3 TOPS → 0.86 TOPS/W. The ANE is genuinely ~20× more energy-efficient for supported ops.
But the GPU wins in these scenarios:
(1) Dynamic shapes — the ANE requires static tensor shapes compiled ahead of time. If your model processes variable-length sequences (NLP, speech), each shape requires a separate compiled graph. The GPU handles dynamic shapes natively via compute shaders. For a chatbot with variable prompt lengths, compiling 100+ shape variants for ANE is impractical.
(2) Unsupported operators — custom attention mechanisms, complex activation functions (Mish, SwiGLU), or novel architectures that aren't in Core ML's ANE op set. The ANE silently falls back to CPU for unsupported ops, creating partition boundaries that destroy throughput. The GPU executes arbitrary compute shaders.
(3) Small batch, high-frequency inference — the ANE has ~0.5-1ms dispatch overhead per inference call. For a real-time audio model running every 10ms, the overhead is 5-10% of the budget. The GPU dispatch overhead is ~0.1ms.
(4) FP16 precision required — the ANE internally quantizes to lower precision for some ops. Medical or financial models requiring strict FP16 numerics must use the GPU.
Napkin Math: ANE: 17.5 TOPS/W (INT8). GPU: 0.86 TOPS/W (INT8 equivalent). ANE is 20× more efficient. But for a model with 30% unsupported ops: ANE path = 5ms (ANE) + 3 handoffs × 1.5ms + 4ms (CPU fallback) = 13.5ms at 2W + 1W (CPU) = 3W → 13.5ms × 3W = 40.5 mJ. GPU path = 12ms (all on GPU) at 5W = 60 mJ. ANE is still cheaper in energy. But if unsupported ops reach 50%: ANE path = 3ms + 5 × 1.5ms + 8ms = 18.5ms at ~3W = 55.5 mJ. GPU: 12ms at 5W = 60 mJ. Nearly equal — and the GPU gives you consistent latency without partition jitter.
📖 Deep Dive: Volume I: HW Acceleration
</details>Interviewer: "Apple's M4 and A-series chips use a unified memory architecture where CPU, GPU, and Neural Engine share the same physical LPDDR5X. Qualcomm's Snapdragon 8 Gen 3 also has shared LPDDR5X. Your colleague says 'they're the same thing — both share memory.' Explain the critical architectural difference and why Apple's approach eliminates a copy overhead that Qualcomm's doesn't."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Both chips share the same DRAM, so there's no difference — data is accessible to all compute units on both." This confuses shared physical memory with a unified memory architecture. Sharing DRAM doesn't mean sharing address spaces.
Realistic Solution: The key difference is address space unification vs shared bus access.
Apple (true unified memory): CPU, GPU, and ANE share a single virtual address space with a unified page table. When the CPU writes a tensor to address 0x1000, the GPU and ANE can read from the same address 0x1000 with zero copies. The hardware coherency protocol ensures all units see the same data. Passing a 50 MB activation tensor from ANE to GPU costs: 0 bytes copied, 0ms latency — it's a pointer handoff.
Qualcomm (shared bus, separate address spaces): The Hexagon NPU, Adreno GPU, and Kryo CPU each have their own memory management units and address spaces. They share the same physical DRAM bus, but a tensor at CPU virtual address 0x1000 is not directly accessible to the NPU. Passing data requires: (1) CPU flushes cache lines to DRAM, (2) runtime copies or remaps the buffer to the NPU's address space, (3) NPU invalidates its cache and reads. For a 50 MB tensor: 50 MB copied (or remapped), 1-3ms latency depending on bus contention.
This matters enormously for multi-accelerator pipelines. A model that runs 5 stages across CPU→ANE→GPU→ANE→CPU on Apple incurs ~0ms transfer overhead. The same pipeline on Qualcomm incurs 4 × 1.5ms = 6ms in transfers — potentially doubling total latency for a 6ms model.
Napkin Math: 5-stage pipeline, 50 MB intermediate tensors. Apple: 5 × 2ms compute + 0ms transfers = 10ms. Qualcomm: 5 × 2ms compute + 4 × 1.5ms transfers = 16ms (60% slower). For a camera pipeline running at 30 FPS (33ms budget), Apple has 23ms headroom, Qualcomm has 17ms. At 60 FPS (16.7ms budget), Qualcomm cannot run this pipeline; Apple can.
📖 Deep Dive: Volume I: HW Acceleration
</details>Interviewer: "You're deploying a 7B parameter LLM on a Samsung Galaxy S24 Ultra (Snapdragon 8 Gen 3, 12 GB RAM, LPDDR5X at 51.2 GB/s). You choose W4A16 quantization — 4-bit weights with 16-bit activations. Walk through the memory savings, the dequantization cost per token, and explain why this asymmetric scheme works better than W4A4 on mobile."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "W4A4 would be even better — quantize everything to 4-bit for maximum compression and speed." W4A4 destroys accuracy on generative models because activations have outlier channels with 100× the magnitude of typical values. Clamping these to 4-bit loses critical information.
Realistic Solution: W4A16 is the sweet spot for on-device LLMs because weights are static (quantized once, stored compressed) while activations are dynamic (computed fresh each token, need precision to preserve outliers).
Memory savings: FP16 weights: 7B × 2 bytes = 14 GB (doesn't fit in 12 GB). INT4 weights: 7B × 0.5 bytes = 3.5 GB. With group quantization (group size 128): add scale + zero-point per group = 7B / 128 × 4 bytes = 218 MB overhead. Total: 3.72 GB — fits with 8.3 GB headroom.
Dequantization cost: Each token generation requires dequantizing all weights from INT4 to FP16 for the matrix-vector multiply. Per weight: one INT4→FP16 conversion = 1 multiply (by scale) + 1 add (zero-point) = 2 FP16 ops. For 7B weights: 14 GFLOPs of dequantization overhead per token. On the Adreno GPU at 4.6 TFLOPS: 14G / 4.6T = 3ms. This is overlapped with the GEMV compute, so effective overhead is ~1-1.5ms per token (memory-bound, not compute-bound).
Why not W4A4: Activation outliers in transformer models follow a power-law distribution. The top 1% of activation channels carry 30-50% of the signal magnitude. INT4 has only 16 discrete levels — it cannot represent both the outlier channels (magnitude ~100) and normal channels (magnitude ~1) without catastrophic clipping. W4A16 preserves these outliers in FP16 while compressing the static, well-distributed weights to INT4.
Napkin Math: FP16: 14 GB (no fit). W4A16: 3.72 GB (fits). Compression: 3.76×. Bandwidth per token: 3.72 GB / 30 GB/s (sustained) = 124ms → 8 tok/s. With speculative decoding (3× acceptance): ~24 tok/s. Dequant overhead: 14 GFLOP / 4.6 TFLOP = 3ms (hidden behind memory latency). W4A4 accuracy: perplexity degrades 15-40% on LLaMA-7B benchmarks. W4A16: perplexity degrades <2%. The activation precision is non-negotiable.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "You're building an iOS app that runs a 1.2 GB CoreML model on an iPhone 15 with 6 GB RAM. During testing, the app works fine in isolation but gets killed when the user switches to Safari and back. Walk through iOS memory pressure levels and calculate at what model size your app starts getting killed on a 6 GB device."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "iOS has 6 GB, my model is 1.2 GB, so I have 4.8 GB free." This ignores that iOS reserves substantial memory for the kernel, system daemons, and the foreground app gets a fraction of total RAM.
Realistic Solution: iOS memory management uses four pressure levels:
(1) Normal — plenty of free pages. No action needed.
(2) Warning (didReceiveMemoryWarning) — the system asks apps to release caches. Your app should drop non-essential buffers.
(3) Critical — the system aggressively kills background apps (most-recently-used first) to free memory.
(4) Jetsam — if the foreground app exceeds its per-process limit, iOS kills it instantly with no warning. No crash log in the normal sense — just a jetsam event.
On a 6 GB iPhone 15: iOS kernel + daemons use ~1.5-2 GB. The foreground app jetsam limit is approximately 2.8-3.2 GB (varies by device state). Background apps share the remaining ~2 GB.
Your app's memory footprint: CoreML model (1.2 GB) + CoreML runtime overhead (~150 MB) + app code + UI (~200 MB) + image buffers (~100 MB) + system frameworks (~300 MB) = ~1.95 GB. This is under the ~3 GB jetsam limit — safe in isolation.
But when the user opens Safari (which caches 500 MB+), then returns to your app: iOS may have evicted your app's mmap'd model pages. Reloading triggers a spike. If your app also allocates a temporary inference buffer (200 MB) during this reload, peak memory hits 2.15 GB + 200 MB = 2.35 GB. Still safe. But add a KV-cache for an LLM conversation (500 MB) and you're at 2.85 GB — dangerously close to jetsam.
Safe model size rule of thumb: On a 6 GB device, keep total app memory under 2.5 GB (with 300-500 MB headroom). Model budget: 2.5 GB - 0.75 GB (overhead) = ~1.75 GB max for the model.
Napkin Math: 6 GB device. OS: ~1.8 GB. Jetsam limit: ~3 GB. App overhead: ~750 MB. Model budget: 3.0 - 0.75 = 2.25 GB (aggressive) or 2.5 - 0.75 = 1.75 GB (safe with headroom). A 2 GB model works in isolation but gets jetsammed under memory pressure. Use
mmap()for weights — the OS can evict and reload pages without killing the app.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your team has a custom sparse attention mechanism that CoreML doesn't support natively — CoreML falls back to CPU for the attention layers, making the model 4× slower than expected. A colleague suggests rewriting the attention in Metal compute shaders to run on the Apple GPU. When is this Metal approach faster than CoreML, and when does it backfire?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Metal gives you direct GPU access, so it's always faster than CoreML for custom ops." Metal compute shaders bypass CoreML's graph optimizer, losing fusion opportunities and ANE delegation for the rest of the model.
Realistic Solution: Metal Performance Shaders (MPS) and raw Metal compute shaders give you programmable GPU access, but the trade-offs are nuanced.
Metal wins when: (1) CoreML falls back to CPU for >20% of compute — the CPU fallback + data transfer overhead exceeds the cost of running the whole model on GPU via Metal. (2) Your custom op has high arithmetic intensity (>10 FLOPs/byte) — the GPU's parallel ALUs shine. (3) You need dynamic control flow within the kernel — Metal supports branching, loops, and atomics that the ANE cannot handle. (4) You're already GPU-bound for rendering — the data is already on the GPU, avoiding a transfer.
Metal backfires when: (1) The custom op is a small fraction of the model — you lose ANE acceleration for the 80% of standard ops that CoreML would have delegated. A model that runs in 5ms on ANE via CoreML might take 15ms entirely on GPU via Metal. (2) The op is memory-bound — the GPU's memory bandwidth (same LPDDR5X) is no better than the ANE's, but the GPU burns 2-3× more power per byte accessed. (3) You create GPU contention with the UI renderer — Metal ML compute and Metal UI rendering share the same GPU, causing frame drops.
Best approach: Use CoreML for the standard ops (delegated to ANE) and inject a custom Metal kernel only for the unsupported attention layer via CoreML's MLCustomLayer protocol. This keeps ANE delegation for 80% of the model while running only the attention on GPU.
Napkin Math: Model: 100 layers, 20 are custom attention. CoreML (CPU fallback): 80 layers ANE (8ms) + 20 layers CPU (40ms) + 20 transfers (30ms) = 78ms. Full Metal GPU: 100 layers GPU (30ms) = 30ms. Hybrid CoreML + Metal custom layer: 80 layers ANE (8ms) + 20 layers GPU (6ms) + 2 transfers (3ms) = 17ms. The hybrid approach is 4.6× faster than pure CoreML fallback and 1.8× faster than full Metal.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "You're deploying a multi-modal model on a Snapdragon 8 Elite using the Qualcomm AI Engine (QNN SDK). QNN can delegate ops to the Hexagon NPU (50 TOPS INT8), Adreno GPU (4.6 TFLOPS FP16), or Kryo CPU. Your model has: Conv2D layers, LayerNorm, GELU, multi-head attention with dynamic sequence lengths, and a final softmax. Build the delegation decision tree and estimate total latency."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Set the QNN backend to 'HTP' (Hexagon Tensor Processor) and let the runtime figure it out." The runtime will silently fall back unsupported ops to CPU, creating partition boundaries that dominate latency.
Realistic Solution: Build an explicit delegation plan by op type:
Hexagon NPU (HTP): Conv2D (native, peak efficiency), MatMul in attention (static shapes only), ReLU/ReLU6, average/max pooling, concatenation, element-wise add/multiply. These ops have hardwired datapaths on the HTP — maximum TOPS/W.
Adreno GPU: GELU activation (not natively supported on HTP — requires exp/tanh approximation), LayerNorm (division by small variance causes precision issues on HTP's INT8 pipeline), multi-head attention with dynamic sequence lengths (HTP requires static shapes; GPU handles dynamic shapes via compute shaders), softmax (exp + division, precision-sensitive).
Kryo CPU: Pre/post-processing (tokenization, image resize, NMS), dynamic control flow (beam search, early exit logic), any ops with complex data-dependent shapes.
Decision tree: For each op: (1) Is it in the HTP supported op list with matching quantization scheme? → HTP. (2) Does it require FP16+ precision or dynamic shapes? → GPU. (3) Does it involve control flow or non-tensor operations? → CPU. (4) Critical: minimize partition boundaries. If a single GPU op sits between two HTP segments, consider approximating it (GELU ≈ x × σ(1.702x)) to keep the graph on HTP.
Napkin Math: Model: 60 layers. 40 Conv/MatMul → HTP (6ms). 12 attention + LayerNorm + GELU → GPU (8ms). 2 pre/post-processing → CPU (1ms). Partition boundaries: HTP→GPU (×2): 2 × 1.5ms = 3ms. GPU→HTP (×2): 2 × 1.5ms = 3ms. GPU→CPU (×1): 1ms. Total: 6 + 8 + 1 + 7 = 22ms. Optimized (approximate GELU on HTP, fuse LayerNorm into GPU attention block): 48 layers HTP (7ms) + 10 layers GPU (6ms) + 2 CPU (1ms) + 2 boundaries (3ms) = 17ms. Saving 3 partition crossings = 5ms = 23% improvement.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "You're adding on-device speech recognition to a messaging app on an iPhone 16 Pro. You benchmark Whisper-tiny (39M params, FP16) and find it takes 180ms to transcribe 1 second of audio on the ANE, but 950ms on the CPU. The real-time factor (RTF) must be below 1.0 for streaming. Calculate the RTF for each, and explain why Whisper's architecture fundamentally conflicts with streaming ASR even when the RTF is acceptable."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Whisper-tiny on ANE has RTF = 0.18, which is well under 1.0, so it works for streaming." RTF < 1.0 means the model processes audio faster than real-time, but Whisper's architecture still prevents true streaming.
Realistic Solution: RTF = processing time / audio duration. ANE: 180ms / 1000ms = 0.18 RTF (5.6× real-time). CPU: 950ms / 1000ms = 0.95 RTF (barely real-time). The ANE path seems great, but Whisper has a fundamental architectural problem for streaming:
Whisper is an encoder-decoder model that processes 30-second chunks. The encoder uses full self-attention over the entire 30-second spectrogram — it cannot produce partial outputs. This means: (1) Minimum latency = 30 seconds — the user must speak for 30 seconds before any transcription appears. (2) Wasted compute on silence — if the user speaks for 3 seconds, Whisper still processes 30 seconds of input (27 seconds of silence/padding). (3) No incremental output — you can't show partial words as the user speaks.
For streaming ASR, you need a model with a causal or chunk-based encoder: (1) RNN-T / Conformer-Transducer (30-80M params) — processes 80ms audio frames incrementally, outputs tokens with ~200ms latency. (2) Chunk-based attention — processes 640ms chunks with look-ahead, outputs with ~800ms latency. (3) CTC-based models — frame-synchronous output, ~160ms latency.
On the iPhone 16 Pro ANE, a 40M-param Conformer-Transducer processes an 80ms frame in ~4ms (RTF = 0.05), with 200ms end-to-end latency. This is the correct architecture for live transcription.
Napkin Math: Whisper-tiny on ANE: 180ms per 1s audio, but 30s minimum input → 30 × 180ms = 5.4s compute for 30s audio. Latency: 30s (unacceptable for messaging). Conformer-Transducer on ANE: 4ms per 80ms frame → RTF = 0.05. Latency: 200ms (excellent). Memory: Whisper-tiny = 39M × 2 = 78 MB. Conformer = 40M × 2 = 80 MB. Similar size, but 150× lower latency for the streaming model.
📖 Deep Dive: Volume I: Network Architectures
</details>Interviewer: "You benchmark your CoreML model on an iPhone 15 Pro and measure 8ms inference latency. But in production, users report the first photo takes 2 seconds to process after opening the app. Subsequent photos are fast. Your QA team confirms: first inference is consistently 100-250× slower than steady-state. What's happening during that first inference, and how do you budget for it?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model file is loading from disk — use a smaller model." Disk I/O is only a fraction of the warm-up cost. The real overhead is compilation and hardware initialization.
Realistic Solution: The first inference triggers a cascade of one-time initialization steps:
(1) Model compilation (~500-1500ms) — CoreML compiles the .mlmodelc into device-specific microcode for the ANE. This involves operator fusion, memory planning, and generating the ANE instruction stream. This is cached after first run, but the cache is invalidated on OS updates.
(2) ANE initialization (~100-300ms) — the Neural Engine is power-gated when idle. First inference wakes it up: clock tree stabilization, SRAM initialization, DMA channel setup.
(3) Memory allocation (~50-200ms) — the runtime allocates intermediate activation buffers, I/O feature buffers, and sets up the memory-mapped weight file. On first allocation, the OS must find and map physical pages.
(4) Weight page faults (~100-500ms) — if weights are mmap'd, the first inference triggers page faults for every weight page accessed. At 4 KB per page, a 200 MB model = 50,000 page faults × 2-10μs each = 100-500ms.
(5) Shader/kernel compilation (~50-100ms on GPU path) — if any ops run on the GPU, Metal shader compilation adds further delay.
Warm-up budget strategy: Call model.prediction(from: dummyInput) during app launch (in applicationDidFinishLaunching) with a zero-filled input tensor. This front-loads all initialization into the launch sequence. Total warm-up: ~1-2 seconds. Hide it behind a splash screen or loading animation. After warm-up, all subsequent inferences run at the benchmarked 8ms.
Napkin Math: Cold first inference: compilation (800ms) + ANE wake (200ms) + allocation (100ms) + page faults for 200 MB model (250ms) + framework init (150ms) = ~1500ms. Warm steady-state: 8ms. Ratio: 1500 / 8 = 187× slower. Warm-up strategy: dummy inference during splash screen. User-perceived first inference: 8ms. Cost: 1.5s added to app launch (hidden behind splash).
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your app runs a 12ms style transfer model on the Apple GPU while simultaneously rendering a complex UI with animations at 60 FPS. Users report dropped frames and UI jank during inference. The GPU has a 16.67ms budget per frame for 60 FPS. Show why the math doesn't add up, and design a scheduling strategy that eliminates frame drops."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "12ms inference + 4ms UI rendering = 16ms total, which fits in the 16.67ms frame budget." This assumes ML inference and UI rendering can be perfectly serialized within a single frame, which ignores GPU command buffer scheduling and preemption behavior.
Realistic Solution: The Apple GPU uses a non-preemptive tile-based deferred renderer. Once a compute or render command buffer starts executing, it runs to completion — the GPU cannot pause your ML inference mid-kernel to service a UI render pass. This means:
Worst case: UI render pass (4ms) is submitted, then ML compute (12ms) is submitted. The ML compute starts immediately after UI render. Next UI frame is due in 16.67ms, but the GPU is busy with ML compute until 4 + 12 = 16ms. The next UI render can't start until 16ms, finishes at 20ms — 3.3ms late, causing a dropped frame. The display shows the previous frame for 33ms (drops to 30 FPS).
Even worse: If ML inference is submitted first (12ms), the UI render pass waits 12ms to start, then takes 4ms = 16ms total. The frame is delivered at 16ms — barely making the deadline. But any variance (thermal throttling, cache miss) pushes it over.
Solution — frame-aware scheduling:
(1) Use MTLCommandBuffer priority — submit UI render passes at .high priority and ML compute at .low. On A15+ GPUs, the hardware scheduler prioritizes high-priority buffers.
(2) Chunk ML compute into small kernels — split the 12ms style transfer into 6 × 2ms kernels. Between each kernel, the GPU can service pending UI render passes. Worst-case UI delay: 2ms (one kernel) instead of 12ms.
(3) Phase-lock to VSync — run ML inference in the first half of the frame (0-8ms), UI rendering in the second half (8-16ms). Use CADisplayLink to synchronize.
(4) Offload to ANE — if the model is CoreML-compatible, run it on the ANE instead, freeing the GPU entirely for UI. ANE and GPU operate in parallel with zero contention.
Napkin Math: Unchunked: 12ms ML + 4ms UI = 16ms. Frame budget: 16.67ms. Margin: 0.67ms (any jitter causes drops). Chunked (6 × 2ms): worst-case UI delay = 2ms. UI render: 4ms. Total frame: 2 + 4 = 6ms. Margin: 10.67ms (safe). ANE offload: GPU has full 16.67ms for UI (4ms render, 12.67ms margin). ML runs on ANE in parallel at 8ms. Zero contention. Frame drops: 0.
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "Your team is building an on-device RAG (Retrieval-Augmented Generation) system on a Galaxy S24 Ultra with 12 GB RAM. The components: a 3B LLM in INT4 (1.5 GB weights), an embedding model for retrieval (100 MB), a vector database of 500K document chunks with 768-dim embeddings, and the retrieved context fed to the LLM. Calculate the total memory footprint and determine if it fits. What's the first thing that has to go if it doesn't?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "1.5 GB + 100 MB + vector DB — should be fine on 12 GB." This forgets the KV-cache, which grows with context length, and the vector DB index overhead, which is much larger than the raw embeddings.
Realistic Solution: Calculate each component:
(1) LLM weights (INT4): 3B × 0.5 bytes = 1.5 GB.
(2) LLM KV-cache: RAG requires long context (retrieved chunks + user query + generation). At 4096 context length, 32 layers, 8 KV heads (GQA), 128 dim, FP16: 2 × 32 × 8 × 128 × 4096 × 2 bytes = 512 MB.
(3) LLM activations + runtime: ~200 MB.
(4) Embedding model: MiniLM-L6 or similar, FP16: 100 MB.
(5) Vector database: 500K chunks × 768 dims × 4 bytes (FP32) = 1.46 GB for raw vectors. HNSW index overhead: ~1.5× raw vectors for graph structure = 2.19 GB total. Alternatively, product quantization (PQ) compresses vectors to 64 bytes each: 500K × 64 = 30.5 MB (with ~5% recall loss).
(6) Retrieved text chunks: 10 retrieved chunks × 512 tokens × 2 bytes = 10 KB (negligible).
(7) Android OS + system: ~3.5 GB.
Unoptimized total: 1.5 + 0.512 + 0.2 + 0.1 + 2.19 + 3.5 = 8.0 GB. Fits in 12 GB with 4 GB headroom. But this leaves little room for other apps — Android will aggressively kill background processes.
Optimized total (PQ vectors + INT8 KV-cache): 1.5 + 0.256 + 0.2 + 0.1 + 0.031 + 3.5 = 5.59 GB. Comfortable with 6.4 GB headroom.
First thing to cut if it doesn't fit: The HNSW index. Replace with product-quantized vectors (1.46 GB → 31 MB) at the cost of ~5% recall degradation. If that's not enough, reduce KV-cache via sliding window attention (512 MB → 128 MB).
Napkin Math: Full HNSW: 2.19 GB. PQ-compressed: 31 MB. Savings: 2.16 GB (98.6% reduction). Recall@10 drops from 98% to 93% — acceptable for most RAG use cases. Total optimized: 5.59 GB on 12 GB device = 46.6% utilization. Safe margin for multitasking.
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "You deploy the same TFLite INT8 model across three Android flagships: Samsung Galaxy S24 (Snapdragon 8 Gen 3), Google Pixel 8 Pro (Tensor G3), and Samsung Galaxy S24 FE (Exynos 2400). Using NNAPI delegation, you measure: Snapdragon — 6ms, Tensor G3 — 9ms, Exynos — 22ms. Same model, same API, 3.7× performance spread. Explain the three layers of fragmentation causing this."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The Exynos just has a slower NPU." While raw NPU performance differs, the 3.7× gap far exceeds the hardware capability difference — the Exynos 2400 NPU is rated at 34.7 TOPS vs Snapdragon's 45 TOPS (only 1.3× difference).
Realistic Solution: Three layers of fragmentation compound to create the gap:
Layer 1: Driver op coverage (2-3× impact). NNAPI defines ~150 operations. Each vendor's driver supports a different subset. Snapdragon's QNN driver supports ~95% of TFLite ops on the Hexagon NPU. Tensor G3's driver supports ~90% on Google's custom NPU. Exynos 2400's driver supports ~70% on the Samsung NPU. The unsupported 30% on Exynos falls back to CPU, creating partition boundaries. If 10 ops fall back: 10 × 1.5ms transfer overhead = 15ms — this alone explains most of the gap.
Layer 2: Quantization scheme mismatch (1.2-1.5× impact). Your model uses per-channel asymmetric INT8 (TFLite default). Snapdragon's NPU natively supports this scheme. Tensor G3 supports it but with per-tensor fast path (slight overhead for per-channel). Exynos 2400's NPU prefers symmetric INT8 — asymmetric requires runtime zero-point adjustment on every MAC, adding ~20% overhead to delegated ops.
Layer 3: Graph optimization maturity (1.1-1.3× impact). Qualcomm has invested years in QNN graph optimization: operator fusion (Conv+BN+ReLU → single kernel), memory planning (reusing activation buffers), and tiling strategies. Samsung's NPU compiler is less mature — fewer fusion patterns, suboptimal tiling, more memory traffic.
Compound effect: Snapdragon: 95% delegation × 1.0 quant overhead × 1.0 compiler efficiency = baseline. Exynos: 70% delegation × 1.2 quant overhead × 1.15 compiler overhead + 15ms transfer penalty. The multiplicative effect of all three layers creates the 3.7× gap.
Napkin Math: Snapdragon: 95% NPU (5ms) + 5% CPU (0.5ms) + 1 transfer (0.5ms) = 6ms. Tensor G3: 90% NPU (6ms) + 10% CPU (1ms) + 2 transfers (2ms) = 9ms. Exynos: 70% NPU (6ms × 1.2 quant × 1.15 compiler = 8.3ms) + 30% CPU (3ms) + 10 transfers (10.7ms) = 22ms. The transfer overhead from poor op coverage dominates on Exynos.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your camera app runs continuous object detection on a Snapdragon 8 Gen 3 phone. After 3 minutes of sustained inference, the SoC junction temperature hits 95°C and the NPU throttles from 45 TOPS to 18 TOPS. Your INT8 model now misses the 33ms frame deadline. Design an adaptive system that switches between INT8 and INT4 precision based on thermal state, and calculate the latency and accuracy at each operating point."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just throttle the frame rate — run at 15 FPS when hot." This degrades the user experience uniformly. Adaptive precision maintains frame rate while gracefully trading accuracy.
Realistic Solution: Design a two-tier inference system with thermal-aware switching:
Tier 1 — Normal thermal (< 85°C): INT8 model at full NPU clock. Model: YOLOv8-S, 11.2M params, INT8. Size: 11.2 MB. Latency at 45 TOPS: ~8ms. Accuracy: 44.9 mAP on COCO. Power: ~3W NPU.
Tier 2 — Throttled thermal (> 85°C): INT4 model at reduced NPU clock. Same architecture, W4A8 quantization. Size: 5.6 MB (weights) + INT8 activations. Latency at 18 TOPS: ~12ms (weights load 2× faster, but reduced TOPS). Accuracy: 42.1 mAP on COCO (2.8 mAP drop). Power: ~1.5W NPU (lower clock + smaller weights).
Switching logic: Monitor thermal_zone0 via Android's ThermalService API. Hysteresis: switch to INT4 at 85°C, switch back to INT8 at 75°C (10°C hysteresis prevents oscillation). Pre-compile both models at app startup — switching is a pointer swap, not a recompilation (~1ms transition).
Thermal feedback loop: INT4 at 1.5W generates less heat → SoC cools → temperature drops below 75°C → switch back to INT8. This creates a natural duty cycle: ~60% INT8, ~40% INT4 during sustained use, maintaining an average of ~43.8 mAP while never missing a frame.
Napkin Math: INT8 at 45 TOPS: 11.2M × 1 byte / (45 × 10⁹ ops/s × efficiency) ≈ 8ms. Fits in 33ms ✓. INT8 at 18 TOPS (throttled): 8ms × (45/18) = 20ms. Still fits 33ms ✓ but leaves only 13ms margin. INT4 at 18 TOPS: weight load halved → ~12ms. Margin: 21ms. Power reduction: 3W → 1.5W. Cooling rate: ~2°C/minute at 1.5W vs heating rate of ~5°C/minute at 3W. Equilibrium: ~80°C with duty cycling. Accuracy: 60% × 44.9 + 40% × 42.1 = 43.8 mAP average (2.4% below peak, but zero dropped frames).
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "When a user takes a Portrait Mode photo on a Pixel 8 Pro (Tensor G3), the pipeline is: capture → depth estimation → person segmentation → bokeh rendering. The user expects the result within 500ms of pressing the shutter. The Tensor G3 has a dual-core TPU at 10 TOPS and an Adreno-class GPU. Break down the latency of each stage and identify the bottleneck."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The ML model (segmentation) is the bottleneck." In modern pipelines, the ML inference is often the fastest stage — the bottleneck is usually ISP processing or the bokeh rendering.
Realistic Solution: Break down each stage:
(1) Capture + ISP processing (~80-120ms): The image signal processor converts raw Bayer sensor data to a 12 MP RGB image. This includes: demosaicing (20ms), noise reduction (30ms), HDR tone mapping (25ms), auto white balance (5ms), lens shading correction (10ms). The ISP is a fixed-function pipeline — you can't speed it up.
(2) Depth estimation (~40-60ms): A lightweight MiDaS-variant model estimates per-pixel depth from the single RGB image (or stereo pair if dual cameras used). 5M params, INT8, on the Tensor G3 TPU. Input: 512×512 downscaled. Output: 512×512 depth map. At 10 TOPS: ~45ms.
(3) Person segmentation (~20-30ms): A DeepLab-variant model produces a binary person/background mask. 2M params, INT8, on TPU. Input: 256×256. Output: 256×256 mask, upsampled to 12 MP via guided upsampling. At 10 TOPS: ~15ms inference + 10ms guided upsampling on GPU.
(4) Bokeh rendering (~100-200ms): Apply depth-dependent Gaussian blur to the background. This is the bottleneck — for each of 12M pixels, compute a variable-radius blur based on depth. Naive implementation: O(pixels × kernel_size²). At 12 MP with average kernel radius 15: 12M × 225 × 3 channels = 8.1 GFLOPs. On GPU at 1 TFLOP: ~8ms. But the variable-radius blur prevents efficient tiling, and memory bandwidth for random access patterns adds 5-10×: ~80-160ms in practice.
Total: 100 + 50 + 25 + 130 = ~305ms. Within the 500ms budget, but the bokeh rendering consumes 43% of the total.
Optimization: Use a layered approach — render bokeh at 1/4 resolution (3 MP) and composite with the sharp foreground at full resolution. Bokeh at 3 MP: ~30ms. Total: 100 + 50 + 25 + 30 = ~205ms.
Napkin Math: ISP: 100ms (fixed). Depth: 45ms (TPU). Segmentation: 25ms (TPU). Bokeh at 12 MP: 130ms (GPU, bottleneck). Bokeh at 3 MP: 30ms (GPU). Total optimized: 205ms (within 500ms budget, 59% margin). The ML stages (depth + segmentation) total 70ms — only 34% of the pipeline. The ISP and rendering dominate.
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "Your app ships a 100 MB INT8 object detection model. You've retrained with 5% more data and fine-tuned the last 10 layers. Pushing a full 100 MB update over cellular wastes user bandwidth. Design a delta update system and calculate the expected patch size. What makes delta compression particularly effective — or tricky — for quantized models?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Use binary diff (bsdiff) on the model files — it'll find the unchanged bytes." Binary diff on quantized models produces surprisingly large patches because quantization parameters (scales and zero-points) shift globally even when only a few layers change, causing byte-level differences throughout the file.
Realistic Solution: Design a layer-aware delta update system:
(1) Layer-level diffing: Compare old and new models layer by layer. Unchanged layers (90% of the model): zero delta. Changed layers (last 10 layers, ~10 MB): extract weight tensors and compute the difference tensor: Δ = W_new - W_old.
(2) Delta compression for INT8 weights: After fine-tuning, most weights change by only ±1-3 quantization levels (the fine-tuning nudges weights slightly). The delta tensor Δ is extremely sparse and low-entropy: ~70% of values are 0, ~25% are ±1, ~5% are ±2-3. This compresses extremely well with entropy coding (Huffman or ANS): 10 MB of deltas → ~0.8-1.2 MB compressed.
(3) Requantization trap: If you requantize the new model independently, the per-channel scales and zero-points may shift even for unchanged layers (because the calibration dataset changed slightly). This makes every byte different. Fix: freeze quantization parameters for unchanged layers. Only recalibrate the fine-tuned layers. This ensures unchanged layers produce zero deltas.
(4) Client-side patching: The client downloads the compressed delta (1 MB), decompresses it, adds Δ to the existing weights for the changed layers, and writes the updated model. Patching time: ~200ms (decompress) + ~50ms (apply delta to 10 MB of weights) = ~250ms.
Napkin Math: Full update: 100 MB over cellular (at 10 Mbps: 80 seconds, costs ~$0.01 at typical data rates). Delta update: changed layers = 10% × 100 MB = 10 MB raw delta. INT8 delta entropy: ~0.8 bits/value (most deltas are 0 or ±1). Compressed: 10 MB × (0.8/8) = 1 MB. Download: 0.8 seconds. Savings: 99% bandwidth reduction. Over 10M users: 100 MB × 10M = 1 PB vs 1 MB × 10M = 10 TB. Server egress savings: ~$50K per update cycle.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "You're building an always-on activity recognition feature (walking, running, cycling, stationary) on a Pixel 8 using the accelerometer at 50 Hz. The feature must run 24/7 without noticeably impacting battery life. Break down the power budget: sensor sampling, inference, and BLE transmission of activity labels to a paired smartwatch. Can you keep total power under 5 mW?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Run inference on every accelerometer sample at 50 Hz." This means 50 inferences per second on the main CPU, which prevents the SoC from entering deep sleep — the power cost of keeping the CPU awake dwarfs the inference cost.
Realistic Solution: Design for duty cycling and offload to the low-power DSP:
(1) Accelerometer sensor: Modern MEMS accelerometers (e.g., Bosch BMI270) consume ~0.9 mW at 50 Hz with a hardware FIFO that buffers 300+ samples. The sensor runs independently — the main CPU stays asleep. Power: 0.9 mW.
(2) Inference on low-power DSP: Batch 100 samples (2 seconds at 50 Hz) into a single inference window. The model: a 1D CNN with 3 layers, ~5K parameters (INT8 = 5 KB). Run on the Tensor G3's low-power "context hub" DSP (not the main CPU). Inference: ~0.5ms per 2-second window. DSP power during inference: ~10 mW. Duty cycle: 0.5ms / 2000ms = 0.025%. Average power: 10 mW × 0.00025 = 0.0025 mW (~0 in practice).
(3) BLE transmission: Send a 1-byte activity label every 2 seconds. BLE 5.0 advertisement: ~10 mW for ~1ms per transmission. Duty cycle: 1ms / 2000ms = 0.05%. Average power: 10 mW × 0.0005 = 0.005 mW.
(4) DSP wake + sleep overhead: Waking the DSP from sleep costs ~0.5 mW average (including sleep-state leakage).
Total: 0.9 + 0.0025 + 0.005 + 0.5 = ~1.41 mW. Well under the 5 mW budget.
Battery impact: Pixel 8 battery: 4575 mAh × 3.87V = 17.7 Wh. At 1.41 mW: 17.7 Wh / 0.00141W = 12,553 hours = 523 days. The feature adds <0.2% to daily battery drain. Imperceptible to the user.
Napkin Math: Sensor: 0.9 mW (always on). DSP inference: 0.0025 mW (duty-cycled). BLE: 0.005 mW (duty-cycled). DSP overhead: 0.5 mW. Total: 1.41 mW. Daily energy: 1.41 mW × 24h = 33.8 mWh. Battery: 17,700 mWh. Drain: 33.8 / 17,700 = 0.19%/day. If you ran inference on the main CPU instead: CPU at 500 mW × 0.025% duty = 0.125 mW (still low), but the CPU wake latency (5ms) and inability to batch means the CPU stays in a shallow sleep state at ~50 mW → 50 mW total = 10× over budget.
📖 Deep Dive: Volume II: Sustainable AI
</details>Interviewer: "Your camera app runs three models simultaneously: face detection, face landmark estimation, and background segmentation. All three use a MobileNetV3-Small backbone (1.5M params, INT8 = 1.5 MB) with different heads. Currently deployed as three independent models totaling 4.5 MB of backbone weights (3 copies). Design a shared-backbone architecture and calculate the memory savings. What are the runtime complications on the Apple ANE?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just share the backbone weights in memory — point all three models to the same weight buffer." This works at the weight level but ignores that each model needs separate activation buffers, and the ANE compiles models as monolithic graphs that can't share subgraphs.
Realistic Solution: Design a two-stage pipeline with explicit backbone sharing:
Architecture: One shared MobileNetV3-Small backbone (1.5 MB) → three lightweight task heads. Face detection head: 200 KB. Landmark head: 150 KB. Segmentation head: 300 KB. Total: 1.5 + 0.2 + 0.15 + 0.3 = 2.15 MB (vs 4.5 + 0.65 = 5.15 MB for three independent models).
Memory savings: Weights: 5.15 MB → 2.15 MB = 58% reduction (3 MB saved). Activations: shared backbone computes features once → one set of intermediate activations (~2 MB) instead of three (~6 MB). Total activation savings: 4 MB. Combined: 7 MB saved.
ANE complications: CoreML compiles each .mlmodelc as an independent execution graph. You cannot share a compiled subgraph between two CoreML models. Options:
(1) Single multi-output model: Compile one CoreML model with three output heads. The backbone executes once, and the ANE runs all three heads sequentially. This is the most memory-efficient but couples the models — you can't update one head without recompiling the whole model.
(2) Backbone + heads as separate models: Run the backbone model once, extract the feature tensor, feed it to three small head models. Problem: each CoreML model invocation has ~0.5ms overhead, and passing intermediate tensors between models requires CPU-side memory copies (~1ms for 2 MB). Total overhead: 3 × (0.5 + 1) = 4.5ms.
(3) Hybrid: Compile backbone + most-critical head (face detection) as one model. Run the other two heads on GPU from the shared feature tensor. Balances efficiency and modularity.
Napkin Math: Three independent models: 3 × 1.5 MB backbone + 3 × 0.2 MB heads = 5.1 MB weights + 3 × 2 MB activations = 11.1 MB total. Shared backbone (single multi-output): 1.5 MB backbone + 0.65 MB heads + 2 MB activations = 4.15 MB total. Savings: 63% (6.95 MB). Latency: independent = 3 × 4ms = 12ms (sequential) or 4ms (parallel on 3 ANE cores). Shared = 3ms backbone + 3 × 0.5ms heads = 4.5ms (always sequential through backbone). Shared is faster if you were running sequentially, slower if you had enough ANE cores for parallelism.
📖 Deep Dive: Volume I: Network Architectures
</details>Interviewer: "ViT-Small (22M params, 4.6 GFLOPs) and MobileNetV3-Large (5.4M params, 0.22 GFLOPs) both achieve ~75% top-1 on ImageNet. On a Snapdragon 8 Gen 3, MobileNetV3 runs in 3ms while ViT-Small takes 35ms — despite ViT having only 21× more FLOPs, it's 12× slower. FLOPs don't predict latency. Explain the architectural reason for this disproportionate slowdown."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "ViT is slower because it has more parameters and FLOPs." The 21× FLOP difference doesn't explain the 12× latency gap — if latency scaled linearly with FLOPs, ViT should take 63ms, not 35ms (it's actually faster than FLOP-linear). The real question is why MobileNetV3 is disproportionately fast, not why ViT is slow.
Realistic Solution: The gap comes from memory access patterns, not arithmetic:
MobileNetV3 (CNN): Depthwise separable convolutions have a highly regular, spatially local memory access pattern. A 3×3 depthwise conv reads a small spatial neighborhood — the data fits in the NPU's on-chip SRAM (typically 1-4 MB). The NPU processes tiles sequentially with near-perfect data reuse. Arithmetic intensity: ~10-50 FLOPs per byte loaded from DRAM. The model is compute-bound on the NPU — DRAM bandwidth is not the bottleneck.
ViT-Small (Transformer): Self-attention computes Q×K^T, which is a dense matrix multiply over the full sequence length. For a 224×224 image with 16×16 patches: sequence length = 196 tokens. The attention matrix is 196×196 = 38,416 elements per head, 6 heads = 230K elements. This matrix doesn't fit in NPU SRAM and must spill to DRAM. Worse, the access pattern is all-to-all — every token attends to every other token, destroying spatial locality. Arithmetic intensity: ~2-5 FLOPs per byte. The model is memory-bandwidth bound.
Additionally, the NPU's fixed-function datapaths are optimized for Conv2D, not MatMul with softmax. ViT's attention layers often fall back to the GPU or CPU, incurring delegation overhead.
Napkin Math: MobileNetV3: 0.22 GFLOPs, compute-bound. NPU at 45 TOPS (INT8): 0.22G / 45T = 0.005ms (arithmetic time). Actual: 3ms (dominated by data movement, but highly optimized tiling). Arithmetic intensity: ~50 FLOPs/byte → compute-bound. ViT-Small: 4.6 GFLOPs. Attention memory: 196 × 196 × 6 heads × 2 bytes = 461 KB per layer × 12 layers = 5.4 MB (exceeds SRAM). DRAM reads: ~50 MB per forward pass. At 51.2 GB/s: 50 MB / 51.2 GB/s = 1ms just for memory. But cache misses and irregular access patterns add 5-10×: ~10ms memory stall. Plus compute: 4.6G / 45T = 0.1ms. Plus delegation overhead (attention on GPU): ~15ms. Total: ~35ms. The memory wall, not the FLOP count, determines transformer latency on mobile.
📖 Deep Dive: Volume I: Network Architectures
</details>Interviewer: "You're building a plant identification app that must work in remote areas with no cellular or WiFi connectivity. The cloud model (ResNet-152, 230 MB, 96% accuracy) is too large for on-device. Design the offline-first ML architecture: what model do you ship, how do you handle the accuracy gap, and what's the total on-device storage budget? Target device: a mid-range phone with 4 GB RAM and 64 GB storage (MediaTek Dimensity 7200)."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just compress the ResNet-152 to fit on-device." Compressing a large model designed for cloud inference produces a worse accuracy/size trade-off than using a model architecturally designed for mobile from the start.
Realistic Solution: Design a tiered offline-first system:
On-device model: EfficientNet-B0 (5.3M params, INT8 = 5.3 MB). Accuracy: 77.1% top-1 on ImageNet, ~85% on a curated plant dataset (narrower domain = higher accuracy). Latency on Dimensity 7200 NPU (~8 TOPS): ~6ms. Fits comfortably in 4 GB RAM.
Accuracy gap mitigation: (1) Domain-specific fine-tuning — train on plant images only (10K species). Domain-specific models outperform general models: 85% → 92% accuracy on plant ID. (2) Confidence thresholding — if model confidence < 80%, show top-3 predictions instead of top-1. Users accept "it might be one of these three" offline. (3) Deferred cloud verification — queue low-confidence predictions. When connectivity returns, send the image to the cloud model for verification. Update the on-device result retroactively.
Storage budget: Model: 5.3 MB. Plant database (10K species × 500-byte description + 50 KB thumbnail): 10K × 50.5 KB = 492 MB. Offline maps for GPS context: ~200 MB. App binary: ~30 MB. Image cache (user photos): ~500 MB. Total: ~1.23 GB out of 64 GB (1.9%).
Sync strategy: When online, download model updates (delta patches: ~500 KB), sync deferred verifications, and update the plant database. Use background fetch to avoid blocking the user.
Napkin Math: Cloud model: 230 MB, 96% accuracy. On-device: 5.3 MB, 92% accuracy (domain-tuned). Size reduction: 43×. Accuracy gap: 4% (mitigated by confidence thresholding + deferred verification). Effective accuracy with deferred verification: ~95% (cloud corrects the 8% of low-confidence predictions within 24 hours). Storage: 1.23 GB / 64 GB = 1.9%. RAM: 5.3 MB model + 50 MB runtime = 55 MB / 4 GB = 1.4%. The offline model is viable on even budget hardware.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "You're deploying a text classification model on the Apple ANE via CoreML. With a fixed input shape of 128 tokens, inference takes 2ms. When you enable dynamic input shapes (1-512 tokens), the same 128-token input takes 6ms — 3× slower on identical hardware and identical input. Why does dynamic shape support cost so much, and when is padding to a fixed shape the better strategy despite wasting compute?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Dynamic shapes are slower because the model has to handle more tokens." The input is the same 128 tokens in both cases — the slowdown comes from the compilation and scheduling path, not the computation itself.
Realistic Solution: The ANE is a fixed-function accelerator with hardwired data paths. Its performance depends on compile-time optimizations that require known tensor shapes:
(1) Memory planning — with fixed shapes, CoreML pre-allocates exact-sized activation buffers at compile time. The ANE's DMA controllers are programmed with fixed addresses and strides. With dynamic shapes, the runtime must allocate buffers at inference time for the actual input size, adding allocation overhead (~0.5ms).
(2) Instruction scheduling — the ANE compiler generates a static instruction schedule (which MAC units fire when, which SRAM banks are read/written) optimized for the exact tensor dimensions. With dynamic shapes, the compiler generates a generic schedule with runtime shape checks and conditional branches — the ANE's simple control unit handles these poorly.
(3) Tiling strategy — for fixed shapes, the compiler computes optimal tile sizes that perfectly partition the tensor across the ANE's processing elements with zero waste. For dynamic shapes, it must use conservative tiling that works for all possible sizes — often leaving processing elements idle.
(4) Kernel fusion — fixed shapes enable aggressive operator fusion (Conv+BN+ReLU as one kernel). Dynamic shapes may prevent fusion if the fused kernel's memory layout depends on the shape.
When to pad: If your input lengths cluster around a few common sizes (e.g., 90% of inputs are 32-128 tokens), compile 3-4 fixed-shape variants (32, 64, 128, 256) and pad each input to the nearest bucket. Wasted compute on padding: at most 2× (a 65-token input padded to 128). But the 3× speedup from fixed shapes more than compensates: 6ms × 0.5 (half wasted) = 3ms dynamic equivalent, vs 2ms fixed. Fixed still wins.
Napkin Math: Fixed 128 tokens: 2ms (optimized schedule, pre-allocated buffers). Dynamic 128 tokens: 0.5ms (allocation) + 0.5ms (shape checks) + 3ms (suboptimal tiling/scheduling) + 2ms (unfused kernels) = 6ms. Bucketed (4 fixed shapes): 128-token input → 128 bucket → 2ms. 65-token input → 128 bucket → 2ms (50% compute wasted, but still 3× faster than dynamic). Break-even: dynamic is better only if inputs vary so widely that you'd need >10 buckets, making the model binary unacceptably large (10 × compiled model size).
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your photo editing app ships a CoreML style-transfer model that works flawlessly on iPhone 15 Pro (8 GB RAM). After launch, crash reports flood in — exclusively from iPhone 11 and iPhone SE 3rd gen users (4 GB RAM). The crash log shows EXC_RESOURCE with type MEMORY and a jetsam event. The model file is only 45 MB. Walk through why a 45 MB model crashes a 4 GB device."
Common Mistake: "The model is 45 MB, the phone has 4 GB — there's plenty of room. It must be a bug in CoreML." This confuses the model file size with the runtime memory footprint, and ignores the drastically lower jetsam limits on older devices.
Realistic Solution: The 45 MB .mlmodelc file is the compressed, on-disk representation. At runtime, CoreML inflates it:
(1) Weight decompression — if the model uses FP16 weights with palettization or LUT compression, CoreML decompresses to full FP16 tensors in memory. 45 MB on disk → ~120 MB in RAM.
(2) Activation buffers — style transfer processes full-resolution images. A 12 MP image at FP16: 12M × 3 channels × 2 bytes = 72 MB input. Intermediate activations for a U-Net style architecture with skip connections: ~4× the input size = 288 MB. Multiple buffers alive simultaneously (encoder + decoder skip connections): peak ~400 MB.
(3) CoreML runtime overhead — compiled graph, memory-mapped buffers, ANE instruction cache: ~80 MB.
Total peak: 120 + 400 + 80 = ~600 MB. On iPhone 15 Pro (8 GB), the jetsam limit is ~3.5 GB — no problem. On iPhone 11 (4 GB), the jetsam limit is ~1.4 GB. The app itself (UI, image cache, system frameworks) uses ~500 MB. That leaves 900 MB for the model — but peak is 600 MB for the model alone, pushing total to 1.1 GB. Add a spike from image decoding and you hit 1.4 GB → jetsam kills the app instantly.
Fix: (1) Downsample input to 1024×1024 before inference (activations drop from 400 MB to ~45 MB). (2) Use MLModelConfiguration with computeUnits = .cpuAndNeuralEngine to avoid GPU memory duplication. (3) Check ProcessInfo.processInfo.physicalMemory at launch and select a model variant: full-res for ≥6 GB, half-res for 4 GB.
Napkin Math: iPhone 11 (4 GB): OS ~1.5 GB, jetsam limit ~1.4 GB, app overhead ~500 MB, model budget = 1.4 - 0.5 = 900 MB. Full-res peak: 600 MB model + 400 MB activations = 1000 MB > 900 MB → crash. Half-res (1024×1024): activations = 1024² × 3 × 2 × 4 = ~25 MB. Peak: 120 + 25 + 80 = 225 MB ✓. The 12× activation reduction from downsampling is the fix, not model compression.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "You ship a CoreML object detection model on iPhone. Xcode benchmarks show 3ms on the A17 Pro Neural Engine. After an iOS update, users report the same model now takes 30ms — a 10× regression. You haven't changed the model or the app. Instruments shows the ANE utilization is 0% and CPU utilization spikes to 100% during inference. What happened?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Apple broke the Neural Engine driver in the iOS update — file a radar and wait." While driver regressions happen, the most common cause is a silent delegation failure that you can diagnose and fix.
Realistic Solution: CoreML's delegation to the ANE is not guaranteed — it's a best-effort optimization that can silently fall back to CPU. Several iOS-update scenarios cause this:
(1) Compiled model cache invalidation — CoreML caches the ANE-compiled model in a device-specific .mlmodelc cache. iOS updates wipe this cache. The first post-update inference triggers recompilation. If the new iOS version's CoreML compiler has stricter op validation, ops that previously delegated to ANE may now fail validation and fall back to CPU. Common triggers: new shape constraints on reshape ops, stricter alignment requirements for conv padding, or deprecated op variants.
(2) Op support regression — Apple occasionally changes which ops the ANE supports between iOS versions. An op like einsum or a specific reduce variant may lose ANE support, causing the entire subgraph containing it to fall back.
(3) computeUnits behavior change — if you specified .all (let CoreML decide), the new iOS version's heuristics may decide CPU is "better" for your model based on updated cost models. The ANE is not used even though it's faster.
Diagnosis: Use MLComputePlan (iOS 17+) to inspect the actual execution plan: which ops run on which compute unit. Compare pre-update and post-update plans. Look for ops that moved from ANE to CPU.
Fix: (1) Pin computeUnits = .cpuAndNeuralEngine to prevent GPU-only fallback. (2) Replace the offending op with an ANE-compatible equivalent (e.g., replace einsum with explicit matmul + transpose). (3) Re-export the model with the latest coremltools version, which generates ops compatible with the new iOS. (4) Ship multiple .mlmodelc variants and select based on iOS version.
Napkin Math: ANE path: 3ms (full delegation, 100% ANE utilization). CPU fallback: model has 80 layers. On A17 Pro CPU (peak single-core ~3.8 GHz): 80 layers × 0.35ms/layer = 28ms. Add CoreML CPU dispatch overhead: ~2ms. Total: 30ms. The 10× regression is exactly the ANE-to-CPU fallback ratio for a model optimized for the ANE's fixed-function pipeline. One unsupported op in the critical path forces the entire graph to CPU because CoreML's graph partitioner decides the transfer overhead of partial delegation exceeds the benefit.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your team ships an on-device LLM chatbot (3B params, INT4, 1.5 GB weights) on a Samsung Galaxy S24 Ultra (Snapdragon 8 Gen 3). Users love it — but after 5 minutes of continuous conversation, the phone hits 45°C skin temperature and Android triggers thermal throttling. The SoC junction temperature is 105°C. The user's phone is in their pocket (no convective cooling). Calculate the thermal budget and design a system that prevents overheating."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Throttle the NPU clock when it gets hot." Reactive throttling is too late — by the time the SoC hits 105°C, the phone skin is already uncomfortably hot and the user has had a degraded experience for minutes.
Realistic Solution: The thermal problem is physics, not software. A phone in a pocket has near-zero convective cooling — heat can only dissipate through conduction to the user's body and radiation, both slow.
Thermal budget: The Galaxy S24 Ultra has a thermal design power (TDP) of ~8.5W sustained with active cooling (in hand, screen on). In a pocket, thermal resistance increases ~3×: sustainable power drops to ~3W before skin temperature exceeds 45°C (the discomfort threshold). LLM decoding on the Hexagon NPU at full clock: ~4.5W. This exceeds the pocket thermal budget by 50%.
Token generation rate at full power: 1.5 GB weights / 30 GB/s (sustained LPDDR5X) = 50ms/token = 20 tok/s at 4.5W. After 3 minutes: junction temp reaches 105°C, NPU throttles to 60% → 12 tok/s at 2.7W. After 5 minutes: skin temp hits 45°C, Android THERMAL_STATUS_SEVERE triggers.
Proactive thermal management:
(1) Token budget system — monitor PowerManager.getThermalHeadroom() (Android 12+). Compute a rolling thermal budget: if headroom < 0.7, reduce generation speed by inserting 50ms delays between tokens. This caps power at ~2.5W (within pocket budget) at the cost of reducing throughput to ~13 tok/s.
(2) Speculative early stopping — if the model's response is likely complete (high probability of EOS in next 5 tokens), stop generating. Average response drops from 150 tokens to 100 tokens: 33% less heat per response.
(3) Conversation cooldown — after 3 consecutive long responses, insert a 10-second "thinking" animation (no inference). This lets the SoC cool ~2°C, preventing cumulative thermal buildup.
(4) Pocket detection — use the proximity sensor + accelerometer (no screen touches, device stationary, proximity sensor covered). When pocket mode is detected, cap NPU to 60% clock preemptively.
Napkin Math: Full power: 4.5W. Pocket thermal limit: ~3W. Overshoot: 1.5W → temperature rises at ~0.5°C/s (junction). From 70°C ambient junction to 105°C throttle: 35°C / 0.5°C/s = 70 seconds to throttle. Throttled power: 2.7W (under budget, but user already felt heat). Proactive cap at 3W: 1.5 GB / (30 GB/s × 0.67 clock) = 75ms/token = 13 tok/s. Temperature stabilizes at ~90°C junction, ~40°C skin. User never feels discomfort. Throughput cost: 20 → 13 tok/s (35% reduction). Acceptable trade-off.
📖 Deep Dive: Volume II: Sustainable AI
</details>Interviewer: "You deploy the same TFLite INT8 face verification model on a Pixel 8 Pro (Tensor G3) and a Samsung Galaxy S23 (Snapdragon 8 Gen 2). On Pixel, the false acceptance rate (FAR) is 0.1% — within spec. On Galaxy S23, the FAR jumps to 1.2% — 12× worse, making the biometric feature unusable. Same model binary, same test images. What's causing the accuracy divergence?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The Snapdragon NPU is less accurate than Google's TPU." Both NPUs compute INT8 arithmetic correctly — the divergence is not in the MAC units but in how the surrounding operations are implemented.
Realistic Solution: Three sources of numerical divergence between SoCs:
(1) Softmax/L2-norm implementation differences — face verification computes cosine similarity between 128-dim embeddings. The final L2-normalization involves a square root and division. The Tensor G3's TPU computes this in FP32 (promoted from INT8 output). The Snapdragon 8 Gen 2's Hexagon NPU uses a fixed-point approximation for the reciprocal square root, introducing ±0.5% error per dimension. Over 128 dimensions, this shifts the cosine similarity by up to 0.02 — enough to push borderline face pairs across the verification threshold (typically set at cosine similarity 0.65).
(2) Requantization rounding — between layers, INT8 activations are requantized (multiply by scale, round, clamp). The rounding mode differs: Tensor G3 uses round-half-to-even (banker's rounding). Hexagon uses round-half-up. For a 50-layer model, the accumulated rounding bias shifts the embedding space systematically.
(3) Operator fusion differences — the Tensor G3 fuses Conv+BN+ReLU into a single kernel with one requantization step. The Snapdragon fuses Conv+BN but applies ReLU separately, adding an extra requantization. Each extra requantization introduces ±0.5 LSB error.
Fix: (1) Run the final embedding normalization and cosine similarity in FP32 on CPU — costs ~0.1ms but eliminates the critical divergence. (2) Calibrate per-device thresholds: ship a calibration set and adjust the verification threshold on first run. (3) Use QNN SDK's "accuracy mode" on Snapdragon, which promotes critical ops to FP16.
Napkin Math: Embedding dimension: 128. Per-dimension error from reciprocal sqrt approximation: ±0.005. Cosine similarity error: √(128) × 0.005 / 128 ≈ 0.0044 per comparison. Verification threshold: 0.65. Genuine pair mean similarity: 0.72 (σ = 0.05). Impostor pair mean: 0.35 (σ = 0.08). At threshold 0.65: FAR on Pixel = P(impostor > 0.65) ≈ 0.01%. With +0.02 systematic shift on Snapdragon: FAR = P(impostor > 0.63) ≈ 0.15%. But outlier impostors in the shifted distribution push FAR to ~1.2%. The 0.02 cosine shift from hardware differences causes a 12× FAR increase because the threshold sits on the steep part of the impostor distribution tail.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "Your photo app has a background ML feature that auto-tags photos using an on-device classification model (MobileNetV3, 5 MB, INT8) on an iPhone 16 (A18 chip). Users report 15% battery drain overnight even when they haven't opened the app. The model inference itself takes only 3ms per photo. With 200 photos to process, that's 600ms of compute — negligible. Where is the battery going?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model is running too many inferences — optimize the model to be faster." 600ms of total compute at ~2W (ANE) = 1.2 joules. iPhone 16 battery is ~67,000 joules. That's 0.002% of battery — the model is not the problem.
Realistic Solution: The battery drain comes from the system costs of waking up to run the model, not the model itself:
(1) Background task wake cost — each BGProcessingTask wake transitions the SoC from deep sleep (~5 mW) to active (~2W). The wake itself takes ~500ms (clock ramp, DRAM refresh, kernel scheduling). If iOS schedules 10 background wake cycles overnight (batching photos): 10 × 500ms × 2W = 10 joules.
(2) Photo library access — reading 200 photos from flash storage. Each photo: ~4 MB HEIF → decode to 12 MP RGB (~36 MB) → resize to 224×224 (~150 KB). The HEIF decode is CPU-intensive: ~50ms per photo at ~3W. Total: 200 × 50ms × 3W = 30 joules. This is 25× more energy than the ML inference.
(3) ANE warm-up — each background wake requires ANE initialization (~200ms at ~3W = 0.6 joules) and model compilation check (~100ms). If the compiled model cache was evicted: full recompilation = 1-2 seconds at ~5W = 5-10 joules.
(4) Screen wake — if the background task triggers a notification ("200 photos tagged!"), the screen lights up for 5 seconds at ~800 mW = 4 joules. Per notification.
(5) Memory pressure cascade — loading 36 MB decoded photos into RAM triggers memory warnings. iOS kills background apps, which later relaunch (each relaunch costs ~2-5 joules). If your photo processing evicts 5 apps: 5 × 3 joules = 15 joules.
Total: 10 + 30 + 5 + 4 + 15 = ~64 joules = 64 / 67,000 = 0.1%. But iOS reports background energy attribution including radio wake-ups, indexing triggered by your metadata writes, and iCloud sync of tags — total attributed: ~10,000 joules = 15%.
Fix: (1) Process photos only when charging (BGProcessingTask with requiresExternalPower = true). (2) Batch all 200 photos in a single wake cycle. (3) Decode photos at 224×224 directly using CGImageSourceCreateThumbnailAtPixelSize — skip the full 12 MP decode. (4) Suppress notifications during background processing.
Napkin Math: ML inference: 200 × 3ms × 2W = 1.2 J (0.002% battery). Photo decode: 200 × 50ms × 3W = 30 J (0.045%). System wake: 10 × 500ms × 2W = 10 J. ANE init: 5 J. Attributed overhead (radio, sync, indexing): ~9,950 J. The ML model is 0.012% of the attributed drain. The photo decode is 0.3%. The system overhead is 99.7%. Optimizing the model saves nothing — optimizing the wake pattern saves everything.
📖 Deep Dive: Volume II: Sustainable AI
</details>Interviewer: "You train a PyTorch model with a custom RotaryPositionEmbedding (RoPE) layer for an on-device LLM. When converting to CoreML with coremltools, the conversion fails with RuntimeError: PyTorch convert function for op 'custom::rope' not found. Your colleague says 'just rewrite the model without RoPE.' Why is that a bad idea, and what are the three correct approaches to handle unsupported ops in CoreML conversion?"
Common Mistake: "Replace RoPE with standard sinusoidal position embeddings — they're equivalent." They are not equivalent. RoPE applies rotation to query and key vectors at each attention layer, enabling relative position encoding that generalizes to unseen sequence lengths. Sinusoidal embeddings are absolute and added once at the input. Removing RoPE degrades long-context performance by 5-15% perplexity and breaks length extrapolation entirely.
Realistic Solution: Three approaches, ordered by preference:
(1) Decompose into supported ops — RoPE is mathematically: split each head into pairs, apply 2D rotation by angle θ = position × frequency. This decomposes into: reshape → slice (even/odd dims) → cos/sin (precomputed) → element-wise multiply → subtract/add → concatenate. All of these ops are CoreML-native. Rewrite the PyTorch forward pass using only these primitives before conversion. Cost: 2-3 hours of engineering. Result: full ANE delegation, zero performance loss.
(2) Register a custom op converter — use coremltools.converters.mil.frontend.torch.register_torch_op to define how custom::rope maps to MIL (Model Intermediate Language) ops. This is a Python function that takes the PyTorch op's inputs and emits MIL operations. The converter runs at conversion time, not inference time — zero runtime overhead.
(3) Use MLCustomLayer (last resort) — implement RoPE as a custom Metal compute shader via CoreML's MLCustomLayer protocol. This runs on the GPU, not the ANE. The layer executes during inference, with ~1ms dispatch overhead per call. For a 32-layer model calling RoPE twice per layer (Q and K): 64 × 1ms = 64ms overhead. Unacceptable for real-time use.
Why approach (1) is best: Decomposed ops run entirely on the ANE with zero overhead. The ANE's fixed-function cos/sin units and element-wise multiply are extremely efficient. Approach (3) forces a GPU round-trip per layer, destroying the ANE pipeline.
Napkin Math: Approach 1 (decomposed): RoPE as 6 native ops per layer. ANE overhead: ~0.05ms per layer × 32 layers = 1.6ms total. Full ANE delegation maintained. Approach 2 (custom converter): same runtime as approach 1 (converter runs at build time). Approach 3 (MLCustomLayer): GPU dispatch per layer: 1ms × 64 calls = 64ms. Plus ANE→GPU→ANE transfers: 64 × 0.5ms = 32ms. Total RoPE overhead: 96ms — more than the rest of the model combined. Approach 1 is 60× faster than approach 3.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your photo app runs a real-time beauty filter (face landmark detection + skin smoothing) on a Snapdragon 8 Elite. The camera preview runs at 30 FPS (33ms budget). The ML model takes 8ms on the Hexagon NPU. The skin smoothing shader takes 5ms on the Adreno GPU. Total: 13ms — well within budget. But users report visible stutter every 2-3 seconds. Instruments shows the frame time spiking to 80ms periodically. What's causing the spikes?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model occasionally takes longer due to thermal throttling." Thermal throttling is gradual — it doesn't cause periodic 80ms spikes every 2-3 seconds. The pattern is too regular.
Realistic Solution: The periodic spikes are caused by garbage collection and memory compaction colliding with the real-time pipeline:
(1) Java/Kotlin GC pauses — Android's ART runtime performs concurrent GC, but the final "pause" phase (marking roots, updating references) stops all threads for 2-10ms. If your camera callback allocates objects (even small ones like Bitmap wrappers, ByteBuffer allocations, or lambda captures), the GC triggers every 2-3 seconds. During the GC pause, the camera frame callback is delayed, and the NPU inference misses its scheduling window.
(2) Camera HAL buffer rotation — the camera subsystem uses a triple-buffer ring. If your processing pipeline holds a buffer too long (because GC paused the callback thread), the camera HAL runs out of buffers and drops a frame. The next frame arrives 33ms late, and your pipeline processes two frames back-to-back: 2 × 13ms = 26ms + the GC pause (10ms) + buffer reacquisition (5ms) = ~54ms for that frame pair.
(3) ISP auto-exposure adjustment — every 2-3 seconds, the ISP recalculates auto-exposure, which briefly stalls the camera pipeline for one frame (~33ms). If this coincides with a GC pause: 33 + 10 = 43ms stall, perceived as a visible hitch.
Fix: (1) Eliminate allocations in the camera callback — pre-allocate all buffers, use ByteBuffer.allocateDirect() (off-heap, no GC). (2) Pin the inference thread to a Performance core with android.os.Process.setThreadPriority(THREAD_PRIORITY_URGENT_DISPLAY). (3) Use ImageReader with maxImages = 4 (quad-buffer) to absorb one dropped frame without visible stutter. (4) Run inference on a dedicated HandlerThread decoupled from the camera callback — if inference is late, display the previous frame's result.
Napkin Math: Normal frame: 8ms NPU + 5ms GPU + 3ms overhead = 16ms < 33ms budget ✓. GC spike: 10ms GC pause + 33ms missed frame + 16ms processing = 59ms (drops 1 frame, visible stutter). With AE adjustment: 33ms ISP stall + 10ms GC + 16ms = 59-80ms (drops 2 frames). Fix: zero-alloc callback + quad-buffer. GC still runs but doesn't affect the camera thread. AE stall absorbed by the extra buffer. Worst case: 16ms + 5ms (buffer swap) = 21ms < 33ms ✓. Zero visible stutter.
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "Your app bundles a 250 MB on-device ML model. Users on cellular connections report that the app download fails or stalls. Your analytics show a 40% drop-off rate during installation on cellular vs 5% on WiFi. The app itself is 30 MB without the model. What are the platform-imposed limits you're hitting, and how do you redesign the delivery?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Compress the model to fit under the limit." Compression alone won't solve the UX problem — users on slow cellular connections will still abandon large downloads.
Realistic Solution: You're hitting multiple platform limits:
(1) iOS App Store cellular download limit — Apple imposes a 200 MB limit for cellular downloads (as of iOS 17). Your 280 MB app (30 MB app + 250 MB model) exceeds this. Users on cellular see "This app is over 200 MB. Connect to Wi-Fi to download." Many users don't have WiFi available and abandon the install.
(2) Android Play Store warning — Google warns users about apps over 150 MB and limits APK size to 150 MB (though AAB can be larger with Play Asset Delivery). A 280 MB download on a 5 Mbps cellular connection takes 7.5 minutes — most users abandon after 60 seconds.
(3) Carrier data caps — a 250 MB model download consumes 2.5% of a 10 GB monthly plan. Users in developing markets with 1-2 GB plans lose 12-25% of their monthly data to one app install.
Redesign — on-demand model delivery:
(1) Ship a tiny base app (30 MB) with a lightweight fallback model (MobileNetV3-Small, 2.5 MB, INT8). This clears all cellular limits.
(2) Download the full model on WiFi using iOS BackgroundAssets framework or Android Play Asset Delivery with on-demand delivery mode. The download happens in the background when WiFi is available.
(3) Progressive model quality — ship three model tiers: Tiny (2.5 MB, 75% accuracy), Medium (25 MB, 88% accuracy), Full (250 MB, 95% accuracy). Download progressively. The user gets immediate functionality with Tiny, and quality improves silently over days.
(4) Model slicing — split the 250 MB model into 10 × 25 MB chunks. Download chunks in priority order (early layers first). The model can run partial inference with the first 5 chunks at reduced accuracy while the rest download.
Napkin Math: Full bundle: 280 MB. Cellular at 5 Mbps: 280 × 8 / 5 = 448 seconds = 7.5 minutes (40% abandon). Tiny app + fallback: 32.5 MB. Cellular: 32.5 × 8 / 5 = 52 seconds = <1 minute (5% abandon). Background WiFi download at 50 Mbps: 250 × 8 / 50 = 40 seconds (invisible to user). Install conversion improvement: 40% → 5% drop-off = 87.5% reduction in lost installs. For 1M install attempts: 350K saved installs.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "Your on-device speech recognition model (Conformer-Transducer, 30M params, INT8) on a Google Pixel 8 Pro (Tensor G3) achieves 5% WER in quiet environments. In a noisy coffee shop (~70 dB ambient), WER degrades to 35% — unusable. The cloud version of the same model handles noise fine because it uses a 200M parameter noise-robust encoder. You can't fit 200M params on-device. How do you fix noise robustness within a 50 MB model budget?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Add a noise suppression preprocessing step before the speech model." A generic noise suppressor (e.g., RNNoise) helps but introduces its own artifacts — it removes noise but also distorts speech harmonics, which can increase WER for certain noise types (babble noise, music).
Realistic Solution: A multi-stage approach within the 50 MB budget:
(1) Learned noise suppression front-end (5M params, INT8 = 5 MB) — train a small U-Net on noisy/clean speech pairs. Unlike generic suppressors, this model is co-trained with the ASR model's feature extractor, so it preserves the acoustic features the ASR model needs. On the Tensor G3 TPU: ~2ms per 80ms audio frame. This alone reduces noisy WER from 35% to ~18%.
(2) Multi-condition training — retrain the Conformer with noise-augmented data (add café noise, street noise, music at 0-20 dB SNR to training data). The model learns noise-invariant representations. No size increase. Reduces noisy WER from 18% to ~12%.
(3) Beamforming with dual microphones — Pixel 8 Pro has 3 microphones. Use a lightweight beamforming algorithm (delay-and-sum, ~0.1ms on CPU) to spatially filter: enhance the direction the user is facing, suppress ambient noise from other directions. Effective SNR improvement: +6-10 dB. Combined with (1) and (2): noisy WER drops to ~8%.
(4) Confidence-based retry — if the ASR model's per-token confidence drops below 0.6 for 3+ consecutive tokens, prompt the user: "I didn't catch that — could you repeat?" This is better than returning garbage transcription.
Total model budget: 30 MB (ASR) + 5 MB (noise suppressor) = 35 MB (under 50 MB budget). Latency: 2ms (suppressor) + 4ms (ASR) = 6ms per 80ms frame → RTF = 0.075.
Napkin Math: Quiet WER: 5%. Noisy WER (baseline): 35%. After noise suppressor: 35% × 0.5 = ~18%. After multi-condition training: 18% × 0.65 = ~12%. After beamforming (+8 dB SNR): 12% × 0.7 = ~8%. Combined: 8% WER in 70 dB noise (vs 35% baseline). Model size: 35 MB. Latency: 6ms per frame (RTF = 0.075). Power: ~1.5W (TPU) for both models. Battery for 1-hour call: 1.5W × 1h = 1.5 Wh / 17.7 Wh = 8.5% battery. Acceptable.
📖 Deep Dive: Volume I: Network Architectures
</details>Interviewer: "You're profiling an on-device LLM (3B params, INT4) on an Apple A18 Pro. Xcode Instruments shows the Neural Engine at 100% utilization during autoregressive decoding, yet you're only getting 15 tokens/second — far below the theoretical maximum. The ANE is rated at 38 TOPS (INT8). At INT4, effective throughput should be even higher. Where are the tokens going?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The ANE is compute-saturated — you need a faster chip." 100% utilization does not mean 100% efficiency. The ANE can be 100% "busy" while spending most of its time waiting for data.
Realistic Solution: The ANE utilization metric counts cycles where the ANE is powered on and dispatched — not cycles where the MAC units are doing useful work. During LLM decoding, the ANE is memory-bandwidth bound, not compute-bound:
(1) The bandwidth wall — autoregressive decoding performs matrix-vector multiplications (GEMV): each token requires loading all 1.5 GB of INT4 weights from DRAM. The A18 Pro's LPDDR5X provides ~55 GB/s peak, ~35 GB/s sustained. Time to load weights: 1.5 GB / 35 GB/s = 43ms per token = 23 tok/s theoretical maximum. You're getting 15 tok/s because of additional overheads.
(2) ANE SRAM thrashing — the ANE has ~32 MB of on-chip SRAM. Each transformer layer's weights are ~47 MB (3B / 32 layers × 0.5 bytes). The weights don't fit in SRAM, so every layer requires streaming from DRAM. The ANE's MAC units stall waiting for weight data — they're "utilized" (powered on, dispatched) but idle (no operands to process).
(3) KV-cache reads — in addition to weights, each token reads the KV-cache for all previous tokens. At 2048 context length with INT8 KV-cache: ~130 MB. This competes with weight reads for DRAM bandwidth: effective bandwidth for weights drops to ~28 GB/s. New time: 1.5 GB / 28 GB/s = 54ms → 18.5 tok/s. With overhead: ~15 tok/s.
(4) Attention compute — the QK^T attention computation is actually compute-bound (not memory-bound), but it's a small fraction of total time. The GEMV weight loading dominates.
Optimization: (1) Reduce model to 2B params — 1 GB weights / 28 GB/s = 36ms → 28 tok/s. (2) Speculative decoding — draft model (200M params) generates 4 candidates, verified in one pass. Effective: 15 × 3.5 = ~52 tok/s. (3) INT4 KV-cache — halves KV bandwidth, freeing ~5 GB/s for weights.
Napkin Math: ANE: 38 TOPS (INT8) = 76 TOPS (INT4 equivalent). Compute per token: 2 × 3B = 6 GFLOP. Compute time: 6G / 76T = 0.08ms (negligible). Memory load: 1.5 GB weights + 130 MB KV = 1.63 GB. At 35 GB/s: 46.6ms. Compute-to-memory ratio: 0.08 / 46.6 = 0.17% — the ANE spends 99.83% of its time waiting for data. "100% utilization" means the ANE is dispatched for 46.6ms per token, but the MAC arrays are active for only 0.08ms. The utilization metric is misleading — effective compute efficiency is 0.17%.
📖 Deep Dive: Volume I: HW Acceleration
</details>Interviewer: "You submit your iOS app to the App Store. It uses an on-device CoreML model for face-based age estimation to apply age-appropriate content filters. Apple rejects it with: 'Guideline 5.1.2 — Data Use and Sharing: Your app collects face data without adequate purpose string.' The model runs entirely on-device and you never send face data to a server. Why was it rejected, and what do you need to change?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "We don't collect face data — the model runs on-device, so there's no privacy issue." Apple's privacy framework doesn't distinguish between on-device processing and server-side collection for certain sensitive data categories. Face geometry is treated as biometric data regardless of where it's processed.
Realistic Solution: Apple's App Store Review Guidelines treat face data as a special category:
(1) NSFaceIDUsageDescription is not enough — this key is for Face ID authentication. Your app uses the TrueDepth camera or Vision framework to detect face geometry, which requires NSCameraUsageDescription with a purpose string that explicitly mentions face analysis. Generic strings like "Camera access needed" are rejected — you must say "This app analyzes your face to estimate age for content filtering."
(2) Privacy Nutrition Label — your App Store Connect listing must declare "Face or Head Data" under "Data Linked to You" or "Data Not Linked to You." Even though the data never leaves the device, Apple requires disclosure of what data types are processed, not just transmitted. Missing this declaration triggers automatic rejection.
(3) Kids category conflict — if your app targets children (content filtering implies this), Apple's Guideline 1.3 (Kids Category) prohibits collecting biometric data from children under 13, even on-device. You must either: (a) remove the age estimation feature for users under 13, or (b) use a non-biometric signal (e.g., parental gate, date-of-birth entry) for the initial age check, and only use the ML model for adult users who consent.
(4) EU Digital Services Act — if distributed in the EU, on-device age estimation for content filtering may require a Data Protection Impact Assessment (DPIA) under GDPR Article 35, even without data transmission.
Fix: Add the correct NSCameraUsageDescription with explicit face analysis mention. Update the Privacy Nutrition Label. Add a consent dialog before first face scan. For kids: use a parental gate instead of face-based age estimation.
Napkin Math: App Store review cycle: ~24-48 hours per submission. Rejection → fix → resubmit: 3-5 days lost. If you're on a launch deadline, this is catastrophic. Prevention: Apple's App Review team publishes a pre-submission checklist. The privacy nutrition label takes 30 minutes to fill out correctly. The
NSCameraUsageDescriptionstring takes 5 minutes. Total prevention cost: 35 minutes. Cost of getting it wrong: 3-5 days + potential PR damage if users see "rejected by Apple" in tech press.
📖 Deep Dive: Volume I: Data Engineering
</details>Interviewer: "Your keyboard app does on-device personalization — it fine-tunes a next-word prediction model (50M params, FP16 = 100 MB) on the user's typing patterns using federated learning on a Pixel 8 (Tensor G3, 128 GB storage). After 3 months, users complain the app is using 4.2 GB of storage. The model is still 100 MB. Where did the other 4.1 GB come from?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The training data (user's typing history) is accumulating." Typing data is small — 10,000 words/day × 5 bytes/word × 90 days = 4.5 MB. That's 0.1% of the bloat.
Realistic Solution: On-device training generates massive hidden storage costs:
(1) Optimizer state — Adam optimizer stores two momentum tensors (m and v) per parameter, each the same size as the weights. 50M params × 2 bytes (FP16) × 2 (m + v) = 200 MB. Plus the gradient tensor: another 100 MB. Total optimizer state: 300 MB.
(2) Checkpoint accumulation — the federated learning protocol saves a checkpoint after each local training round (typically daily). Each checkpoint: 100 MB (weights) + 300 MB (optimizer state) = 400 MB. After 90 days with no cleanup: 90 × 400 MB = 36 GB. But your app keeps only the last 10 checkpoints for rollback: 10 × 400 MB = 4 GB. This is the primary bloat source.
(3) Training data buffer — the on-device training pipeline caches preprocessed training examples (tokenized, batched, shuffled) in a SQLite database. 90 days of typing at 10K words/day, tokenized with vocabulary indices (4 bytes each) plus context windows (128 tokens per example): ~2M examples × 128 × 4 bytes = 1 GB. With SQLite overhead and WAL journal: ~1.2 GB.
(4) Gradient accumulation logs — for federated learning, the app stores gradient updates to upload during the next FL round. If the server round hasn't happened (user was offline), gradients accumulate: 100 MB per round × 5 pending rounds = 500 MB.
Total: 100 MB (model) + 4 GB (checkpoints) + 1.2 GB (training cache) + 500 MB (pending gradients) = ~5.8 GB. Your reported 4.2 GB is after some automatic cleanup.
Fix: (1) Keep only 2 checkpoints (current + previous): 800 MB. (2) Save only weight diffs from base model (delta checkpoints): ~20 MB each → 40 MB. (3) Use INT8 optimizer state: 150 MB (vs 300 MB). (4) Cap training cache at 100 MB with FIFO eviction. (5) Compress pending gradients with top-k sparsification (keep top 1% of gradients): 500 MB → 5 MB.
Napkin Math: Unoptimized: 100 + 4000 + 1200 + 500 = 5.8 GB. Optimized: 100 MB (model) + 40 MB (2 delta checkpoints) + 150 MB (INT8 optimizer) + 100 MB (capped cache) + 5 MB (sparse gradients) = 395 MB. Reduction: 93%. The key insight: on-device training's storage cost is dominated by checkpoints and optimizer state, not the model or training data.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "You're building a real-time AR app on an iPhone 15 Pro (A17 Pro) that overlays virtual objects on camera frames. Your CoreML pose estimation model benchmarks at 6ms ± 0.2ms in isolation. In production, the same model shows 6ms median but with a long tail: P95 = 12ms, P99 = 22ms — a 3.7× spread. The P99 spikes cause visible AR object jitter. Identify the sources of timing variance and design a system that guarantees sub-10ms P99."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model is non-deterministic — run it multiple times and average." The model computation is deterministic. The variance comes from system-level interference that's invisible in isolated benchmarks.
Realistic Solution: Five sources of timing jitter on mobile, ordered by impact:
(1) DVFS (Dynamic Voltage and Frequency Scaling) — the A17 Pro adjusts ANE clock frequency based on thermal state and power budget. At full clock: 6ms. At 70% clock (after 30 seconds of sustained inference): 8.6ms. The clock changes happen mid-inference, causing unpredictable latency. Contributes ~2-3ms variance.
(2) Memory bandwidth contention — the ANE shares LPDDR5X bandwidth with the GPU (rendering AR scene), ISP (processing camera frames), and display controller (refreshing screen). During a GPU render burst, available bandwidth drops 30-40%. A memory-bound model layer that normally takes 1ms takes 1.5ms. Across 20 memory-bound layers: +10ms. This is the primary P99 culprit.
(3) OS scheduling interference — iOS kernel tasks (memory compaction, spotlight indexing, iCloud sync) can preempt the inference thread for 1-5ms. The ANE itself isn't preempted, but the CPU thread that dispatches work to the ANE and reads results is.
(4) Thermal throttling steps — iOS throttles in discrete steps, not continuously. A step change from "nominal" to "fair" thermal state drops ANE throughput by ~20% instantly, causing a latency jump.
(5) CoreML graph scheduling — CoreML's internal scheduler may repartition the execution graph between ANE and CPU based on runtime heuristics. If it decides to move one layer to CPU mid-session, that layer goes from 0.3ms (ANE) to 3ms (CPU).
Guaranteed sub-10ms P99 design:
(1) Dedicated QoS thread — run inference on a thread with QOS_CLASS_USER_INTERACTIVE priority. This gets highest scheduling priority, minimizing OS preemption.
(2) Frame budget reservation — allocate 10ms for inference in the frame budget. If inference exceeds 8ms (measured via mach_absolute_time), skip the current frame's inference and reuse the previous pose estimate. The AR object uses IMU-based prediction for one frame — imperceptible.
(3) Thermal-aware model switching — monitor ProcessInfo.thermalState. At .serious or above, switch to a lighter model (3ms at throttled clocks). Two models, hot-swappable.
(4) Pin to ANE — use MLModelConfiguration with computeUnits = .cpuAndNeuralEngine to prevent CoreML from repartitioning to GPU (which would contend with AR rendering).
Napkin Math: Jitter sources: DVFS (±2ms), bandwidth contention (±5ms at P99), OS scheduling (±3ms at P99), thermal steps (±3ms), graph repartition (±3ms). Worst-case stack: 6 + 2 + 5 + 3 + 3 + 3 = 22ms (matches observed P99). With mitigations: QoS thread eliminates OS scheduling (±0ms). ANE pinning eliminates repartition (±0ms). Thermal switching caps DVFS impact (±1ms). Bandwidth contention remains (±5ms at P99). New P99: 6 + 1 + 5 = 12ms. With frame skip at 8ms threshold: effective P99 = 8ms (skip rate ~5%, imperceptible with IMU prediction).
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "Your on-device keyboard prediction model (LSTM, 20M params) on a Snapdragon 8 Gen 3 learns from user typing to improve suggestions. A security researcher demonstrates that by observing the model's top-5 prediction probabilities over 1000 queries, they can reconstruct fragments of the user's private text — including passwords typed into non-password fields and private messages. The model never leaves the device. How is private data leaking through predictions?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model is on-device, so data can't leak." On-device doesn't mean private. The model's outputs (predictions) are observable by any app using the keyboard, and those outputs encode information about the training data.
Realistic Solution: This is a model inversion / membership inference attack on a personalized on-device model:
(1) Memorization in small models — a 20M parameter LSTM trained on one user's data (relatively small corpus: ~50K unique sentences) memorizes specific sequences. If the user types "my password is" frequently (e.g., in notes, messages), the model learns to predict the actual password with high confidence. Any app that reads the keyboard's suggestion bar can observe this.
(2) Prediction probability leakage — the top-5 predictions with confidence scores reveal the training distribution. If "meeting with [CEO_NAME]" appears with 0.85 confidence, an attacker knows the user frequently types that CEO's name — leaking business relationships.
(3) Gradient leakage in federated learning — if the keyboard participates in federated learning, the gradient updates sent to the server can be inverted to reconstruct training examples. A 20M parameter gradient update contains enough information to reconstruct ~100 training sentences with high fidelity.
Mitigations:
(1) Differential privacy (DP) in training — add calibrated Gaussian noise to gradients during on-device training. With ε = 8 (moderate privacy): prediction quality drops ~3%, but memorization of specific sequences drops >90%. The model learns general patterns ("meeting with [PERSON]") but not specific names.
(2) Prediction filtering — never suggest tokens that appear fewer than 5 times in the user's history. This prevents one-off sensitive entries (passwords, SSNs) from appearing in suggestions. Maintain a frequency counter per token.
(3) Sensitive field detection — detect password fields (isSecureTextEntry), credit card fields, and SSN fields via input type attributes. Disable personalized predictions in these fields — use only the base (non-personalized) model.
(4) Temporal decay — exponentially decay the influence of old training data. Tokens typed >30 days ago have their contribution reduced by 90%. This limits the window of vulnerability.
Napkin Math: LSTM with 20M params, trained on 50K sentences. Memorization capacity: a 20M param model can memorize ~200K tokens verbatim (1 param ≈ 10 bits, 200K tokens × 100 bits/token = 20M × 10 bits). User types ~10K words/day × 90 days = 900K tokens. The model can memorize ~22% of the user's typing history verbatim. With DP (ε = 8): effective memorization drops to <2%. Prediction quality (perplexity): base model 45, personalized 32, personalized + DP 35. The 3-point perplexity cost buys 10× privacy improvement.
📖 Deep Dive: Volume I: Data Engineering
</details>Interviewer: "Your ML-powered camera app works perfectly on the Samsung Galaxy S24 (Snapdragon 8 Gen 3, 8 GB RAM) but crashes on the Samsung Galaxy A15 (MediaTek Helio G99, 4 GB RAM) within 30 seconds of opening the camera. The model is a 15 MB INT8 segmentation model. Logcat shows java.lang.OutOfMemoryError followed by a native crash in the TFLite runtime. The model alone is only 15 MB. Why does a 15 MB model crash a 4 GB phone?"
Common Mistake: "The model is too big for the phone — use a smaller model." 15 MB is tiny. The crash has nothing to do with the model's file size.
Realistic Solution: The 15 MB model file expands dramatically at runtime, and the Galaxy A15's memory budget is far smaller than you think:
(1) Android memory budget on 4 GB — Android reserves ~1.5 GB for the OS and system services. The per-app heap limit on budget phones is typically 256 MB (Java heap) + 512 MB (native heap). Total app budget: ~768 MB.
(2) Camera buffer allocation — your app opens the camera at 1080p preview (1920×1080). Android's Camera2 API allocates: 3 preview buffers × 1920 × 1080 × 1.5 (YUV_420_888) = 9.3 MB. Plus 1 capture buffer at 12 MP: 12M × 1.5 = 18 MB. Plus the ImageReader Java wrapper objects.
(3) TFLite runtime expansion — the 15 MB INT8 model at runtime: weights stay at 15 MB (mmap'd). But activation tensors for a segmentation model processing 512×512 input: ~80 MB (multiple intermediate feature maps alive simultaneously due to skip connections). TFLite's memory planner pre-allocates the peak activation footprint at model load time.
(4) Image preprocessing — converting the camera YUV frame to RGB, resizing to 512×512, normalizing to float: the intermediate Bitmap objects consume 1920 × 1080 × 4 (ARGB) = 8.3 MB per frame. If the GC doesn't collect the previous frame's bitmap before the next arrives (30 FPS = 33ms between frames), 2-3 bitmaps accumulate: 25 MB.
(5) The cascade — 15 MB (model) + 80 MB (activations) + 27 MB (camera buffers) + 25 MB (bitmaps) + 100 MB (app UI + system frameworks) = 247 MB. This is at the 256 MB Java heap limit. One more bitmap allocation triggers OutOfMemoryError. The TFLite native runtime then tries to access freed memory → native crash.
Fix: (1) Process at 256×256 instead of 512×512 — activations drop from 80 MB to 20 MB. (2) Use Bitmap.recycle() immediately after preprocessing. (3) Use TFLite's allowBufferHandleOutput to avoid CPU-side tensor copies. (4) Check ActivityManager.getMemoryClass() at startup and select model variant accordingly.
Napkin Math: Galaxy S24 (8 GB): per-app budget ~1.5 GB. Runtime footprint: 247 MB. Headroom: 1.25 GB ✓. Galaxy A15 (4 GB): per-app budget ~768 MB. Runtime footprint: 247 MB. Headroom: 521 MB. But add a few WebViews (50 MB each), ad SDK (30 MB), analytics (20 MB): 247 + 100 + 30 + 20 = 397 MB. Headroom: 371 MB. One memory spike from camera auto-focus (allocates temporary buffers): +200 MB → 597 MB > 512 MB native limit → crash. The fix (256×256 input): activations drop 60 MB, bitmaps drop 18 MB. New total: 169 MB. Headroom: 599 MB. Survives the spike.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "You deploy an INT4 quantized LLM (Gemma-2B) across three flagship phones: iPhone 16 Pro (A18 Pro), Galaxy S24 Ultra (Snapdragon 8 Gen 3), and Pixel 9 Pro (Tensor G4). The same prompt produces noticeably different outputs on each device — not just minor token differences, but semantically different responses. All three use the same INT4 weight file. How can identical weights produce different text on different SoCs?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "INT4 is INT4 — the math is the same everywhere." INT4 quantization is not a single standard. The dequantization formula, grouping strategy, and accumulation precision vary between runtimes and hardware.
Realistic Solution: Four levels of divergence compound to produce different outputs:
(1) Dequantization precision — INT4 weights are dequantized to a higher precision for computation: float_weight = (int4_weight - zero_point) × scale. On the A18 Pro ANE, this dequantization happens in FP16. On the Snapdragon Hexagon NPU, it happens in FP16 but with a different rounding mode for the scale multiplication. On the Tensor G4 TPU, dequantization uses BF16 (different mantissa bits than FP16). Each produces slightly different floating-point weights.
(2) Accumulation precision — the matrix-vector multiply accumulates partial sums. The ANE accumulates in FP16 (10-bit mantissa). The Hexagon accumulates in FP32 internally, then truncates to FP16 for output. The Tensor G4 accumulates in FP32. For a 2048-dim dot product, FP16 accumulation loses ~0.1% of the signal vs FP32 accumulation. This shifts logits by 0.01-0.05.
(3) Softmax temperature amplification — the logit differences from (1) and (2) are small (~0.05). But softmax with temperature 1.0 exponentiates these differences: if logit_A = 5.00 and logit_B = 4.95 on device 1, but logit_A = 4.98 and logit_B = 4.97 on device 2, the probability ratios change dramatically: device 1: P(A)/P(B) = e^0.05 = 1.051. Device 2: P(A)/P(B) = e^0.01 = 1.010. With sampling (temperature > 0), these probability shifts cause different tokens to be selected.
(4) RNG implementation — if using top-p or top-k sampling, the random number generator seed and implementation differ across platforms. Even with the same seed, different RNG algorithms (Philox on CUDA-derived code, Mersenne Twister on CPU) produce different random sequences.
Fix: (1) Use greedy decoding (temperature = 0) for deterministic output — eliminates RNG divergence. (2) Run softmax and final sampling in FP32 on CPU across all platforms. (3) Use a platform-independent quantization format (e.g., GGUF with explicit dequantization spec). (4) Accept non-determinism as inherent to quantized LLMs and evaluate quality statistically, not per-output.
Napkin Math: FP16 vs BF16 mantissa: FP16 has 10 bits (precision ~0.001), BF16 has 7 bits (precision ~0.008). Per-weight dequantization error: ±0.004 (FP16) vs ±0.03 (BF16). Over a 2048-dim dot product: accumulated error = √2048 × per-weight error. FP16: √2048 × 0.004 = 0.18. BF16: √2048 × 0.03 = 1.36. Logit shift: 0.18 vs 1.36. Softmax probability shift for top token: e^0.18 = 1.20 vs e^1.36 = 3.90. At temperature 0.7: amplified further. The BF16 path produces 3× more divergent sampling probabilities than FP16, explaining why Tensor G4 outputs differ most from the other two.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "Your note-taking app has an on-device semantic search feature on a Pixel 9 (Tensor G4, 12 GB RAM). Users index ~10K notes with 384-dim embeddings using an HNSW index. A user reports that searching for 'vacation photos from Italy' returns their tax documents instead. The embedding model is correct — you verified the embeddings are semantically meaningful. What's wrong with the vector search?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The embedding model is bad — fine-tune it on the user's data." You already verified the embeddings are correct. The bug is in the index, not the embeddings.
Realistic Solution: Three failure modes in on-device HNSW that produce wrong results:
(1) Distance metric mismatch — the embedding model was trained with cosine similarity, but the HNSW index was configured with L2 (Euclidean) distance. For normalized embeddings, cosine and L2 are equivalent. But if the embeddings are not L2-normalized before insertion (a common oversight), L2 distance favors vectors with smaller magnitude regardless of direction. Tax documents with short, factual text produce shorter embeddings (lower magnitude) than descriptive vacation notes. L2 distance ranks the tax docs closer to the query because they're closer to the origin, not because they're semantically similar.
Fix: Either normalize all embeddings to unit length before insertion, or configure the index to use cosine distance (inner product on normalized vectors).
(2) HNSW graph corruption from concurrent writes — the user adds notes while a search is in progress. HNSW's graph structure is not thread-safe by default. A concurrent insert can corrupt the neighbor lists of existing nodes, creating "shortcuts" in the graph that skip over the correct nearest neighbors. The search traverses the corrupted graph and terminates at a local minimum that's far from the true nearest neighbor.
Fix: Use a read-write lock. Batch inserts during idle time. Or use a concurrent-safe HNSW variant (e.g., hnswlib with allow_replace_deleted).
(3) Quantization of stored vectors — to save memory, you quantized the stored vectors from FP32 to INT8 (10K × 384 × 4 bytes = 15 MB → 10K × 384 × 1 byte = 3.75 MB). Scalar quantization with a global min/max across all dimensions loses per-dimension dynamic range. If dimension 42 has range [-0.01, 0.01] and dimension 100 has range [-5.0, 5.0], the global quantization allocates most of the 256 INT8 levels to dimension 100's range, leaving dimension 42 with only 1-2 distinct quantized values. Dimensions with small range (which may be the most discriminative) lose all information.
Fix: Use per-dimension quantization (separate scale/zero-point per dimension) or product quantization (PQ), which handles heterogeneous dimension ranges.
Napkin Math: 10K notes, 384-dim, FP32: 10K × 384 × 4 = 15 MB (fits easily in 12 GB). With L2 distance on unnormalized embeddings: query embedding magnitude = 12.5, vacation note magnitude = 15.2, tax doc magnitude = 8.1. L2 to query: vacation = √(Σ(q-v)²) ≈ 18.3, tax = √(Σ(q-t)²) ≈ 14.9. Tax doc is "closer" in L2 despite being semantically wrong. With cosine distance: vacation cosine = 0.89, tax cosine = 0.23. Correct ranking. The normalization step costs: 10K × 384 multiplies + 10K square roots = <1ms. Skipping it breaks the entire search.
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "Your app uses an on-device ML model to auto-generate image descriptions for a social media feed on an iPhone 15 (A16 Bionic). Blind users relying on VoiceOver report that the app is unusable — VoiceOver reads 'image' for every photo instead of the ML-generated description. The model works correctly when you test it. Sighted users see the descriptions. What's happening?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "VoiceOver can't read ML-generated text." VoiceOver reads whatever is in the accessibilityLabel property. The issue is timing, not capability.
Realistic Solution: The ML description is generated asynchronously, but VoiceOver reads the accessibility label at the moment the cell appears on screen — before the ML model has finished processing:
(1) Race condition — the feed loads, cells appear, VoiceOver immediately reads accessibilityLabel. Your code sets accessibilityLabel = "image" as a placeholder, then dispatches ML inference asynchronously. The model takes 50ms per image. VoiceOver reads the placeholder in <10ms. By the time the model returns the description, VoiceOver has already moved to the next cell.
(2) Cell reuse — UITableView/UICollectionView reuses cells. When a cell is reused, the previous image's ML description may briefly remain as the accessibilityLabel before the new image's description is ready. VoiceOver reads the stale description — the user hears a description of a completely different image.
(3) Missing accessibility notification — even if you update accessibilityLabel after inference completes, VoiceOver doesn't re-read it unless you post UIAccessibility.Notification.layoutChanged. Without this notification, the updated label is invisible to VoiceOver until the user manually navigates away and back.
Fix: (1) Pre-compute descriptions during feed loading (before cells appear). Cache descriptions keyed by image URL. (2) If the description isn't ready, set accessibilityLabel = "Image loading, please wait" and post UIAccessibility.Notification.layoutChanged when the description arrives. (3) Prefetch descriptions for the next 5 cells using UICollectionViewDataSourcePrefetching. (4) On cell reuse, immediately clear the accessibility label to prevent stale reads.
Napkin Math: VoiceOver read delay: ~200ms after cell appears. ML inference: 50ms per image. If inference starts when the cell appears: description ready at 50ms < 200ms VoiceOver delay → should work. But: inference is queued behind other images. With 10 visible cells, queue depth = 10. Serial processing: 10 × 50ms = 500ms for the last cell. VoiceOver reads cell 10 at 200ms, but its description arrives at 500ms — 300ms too late. Fix: prefetch 15 cells ahead. At scroll speed of 5 cells/second: prefetch gives 3 seconds lead time. 15 × 50ms = 750ms to process all prefetched cells. Headroom: 2.25 seconds. All descriptions ready before VoiceOver needs them.
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "Your video calling app applies a real-time background replacement model (DeepLabV3, 2.1M params, INT8) on a MediaTek Dimensity 9300 (NPU at 37 TOPS). At 720p 30 FPS, the model runs in 12ms per frame — within the 33ms budget. But users report the video drops to 15 FPS during calls. The model isn't the bottleneck. What's consuming the other 21ms per frame?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model is taking longer than benchmarked due to thermal throttling." 12ms × 2 (throttled) = 24ms, which still fits in 33ms. Throttling alone doesn't explain 15 FPS.
Realistic Solution: The full video pipeline has stages beyond ML inference that consume the frame budget:
(1) Camera frame acquisition + ISP (~5ms) — the camera HAL delivers a YUV frame. The ISP applies noise reduction, auto-exposure, and auto-white-balance. This is fixed-function hardware — you can't skip it.
(2) Color space conversion (~3ms) — converting YUV_420 to RGB for the ML model. On CPU: 720 × 1280 × 3 bytes × 2 (read + write) = 5.5 MB of memory traffic. At 17 GB/s LPDDR5: 0.3ms for memory, but the CPU conversion kernel is inefficient: ~3ms.
(3) ML inference (~12ms) — segmentation on NPU. Produces a 720p binary mask.
(4) Mask refinement (~4ms) — the raw segmentation mask has jagged edges. A guided filter or bilateral filter smooths the boundary. On GPU: ~4ms for 720p.
(5) Background compositing (~3ms) — alpha-blend the foreground (user) with the replacement background using the refined mask. On GPU: 720p × 3 channels × 2 (foreground + background) = 5.5 MB. At GPU memory bandwidth: ~3ms.
(6) Video encoding (~8ms) — the composited frame must be encoded to H.264/H.265 for transmission. The hardware video encoder takes ~8ms per frame at 720p.
(7) WebRTC packetization (~2ms) — the encoded frame is packetized for network transmission.
Total: 5 + 3 + 12 + 4 + 3 + 8 + 2 = 37ms > 33ms budget → drops to ~27 FPS. But the pipeline is serial — each stage waits for the previous. Any jitter pushes individual frames to 45-50ms, and the frame rate averages to ~15 FPS due to frame skipping.
Fix: (1) Pipeline parallelism — while frame N is in ML inference, frame N-1 is in compositing, and frame N-2 is in encoding. Three-stage pipeline: throughput = max(stage latency) = 12ms → 83 FPS theoretical. (2) GPU color conversion — replace CPU YUV→RGB with a GPU compute shader: 3ms → 0.5ms. (3) Process at 360p — downsample for ML, upsample mask to 720p. Inference: 12ms → 4ms. (4) Combined pipeline: max(5, 0.5, 4, 4, 3, 8, 2) = 8ms per frame → 30+ FPS sustained.
Napkin Math: Serial pipeline: 5 + 3 + 12 + 4 + 3 + 8 + 2 = 37ms (27 FPS, drops to 15 with jitter). Pipelined at 360p: max stage = 8ms (encoding). Throughput: 1000 / 8 = 125 FPS (capped at 30 by camera). Latency: 5 + 0.5 + 4 + 4 + 3 + 8 + 2 = 26.5ms (one frame of pipeline delay, acceptable for video calls). The encoding stage, not the ML model, is the true throughput bottleneck.
📖 Deep Dive: Volume I: Model Serving
</details>Interviewer: "Your app performs on-device fine-tuning of a 100M parameter image classifier on a Samsung Galaxy S24 (Snapdragon 8 Gen 3, 12 GB RAM) to personalize for the user's specific objects. After 50 fine-tuning steps, the model's accuracy on the user's objects improves from 60% to 92%. But the model's accuracy on the original 1000 ImageNet classes drops from 85% to 12% — catastrophic forgetting. The base model is effectively destroyed. How do you fine-tune on-device without corrupting the base model?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Use a lower learning rate to prevent catastrophic forgetting." A lower learning rate slows forgetting but doesn't prevent it — after enough steps, the model still drifts. And on mobile, you want fast adaptation (few steps), which requires a higher learning rate.
Realistic Solution: Three approaches for safe on-device fine-tuning, ordered by memory efficiency:
(1) LoRA (Low-Rank Adaptation) — freeze all base model weights. Add small rank-4 adapter matrices to each attention/linear layer. Only train the adapters: 100M param model with rank-4 LoRA → ~400K trainable parameters (0.4% of total). Memory for adapter weights: 400K × 4 bytes (FP32 for training) = 1.6 MB. Optimizer state (Adam): 2 × 1.6 MB = 3.2 MB. Total training overhead: 4.8 MB. The base model is never modified — catastrophic forgetting is impossible by construction. At inference: merge adapters into base weights (one-time 50ms operation) or run as a side-branch (adds ~0.5ms latency).
(2) Head-only fine-tuning — freeze the entire backbone, replace the classification head with a new head for the user's classes. Train only the head: ~50K parameters for 10 user classes. Training memory: ~200 KB. Faster than LoRA but less expressive — can't adapt feature extraction to the user's domain.
(3) Elastic Weight Consolidation (EWC) — add a regularization term that penalizes changes to weights that are important for the original task (measured by the Fisher information matrix). Allows full fine-tuning while preserving base knowledge. But: computing the Fisher matrix requires a forward pass over a calibration set (100 images × 100M params = ~400 MB of gradient storage). Too expensive for mobile.
On-device LoRA implementation:
Storage: base model (100 MB, read-only, mmap'd) + adapter (1.6 MB, writable). The base model is never written to — even if the app crashes mid-training, the base model is intact. The adapter is saved after each training step (1.6 MB write, ~5ms on UFS 4.0). If the adapter is corrupted, delete it and fall back to the base model.
Napkin Math: Full fine-tuning: 100M params × 4 bytes (FP32) = 400 MB weights + 800 MB optimizer = 1.2 GB training memory. Risk: base model corrupted. LoRA (rank 4): 400K params × 4 bytes = 1.6 MB weights + 3.2 MB optimizer = 4.8 MB training memory. Risk: zero (base model frozen). Memory reduction: 250×. Accuracy on user objects: full fine-tuning 92%, LoRA 89% (3% lower but base model preserved). Accuracy on original classes: full fine-tuning 12% (destroyed), LoRA 84% (1% drop from base). LoRA is the only viable approach for on-device fine-tuning.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "Your app uses an on-device LLM (1.5B params, INT4, 750 MB) for a smart reply feature on an iPhone 16 Pro (A18 Pro). Users complain that the app takes 4.5 seconds to launch — the App Store average for your category is 1.5 seconds. Profiling shows 3 seconds are spent in MLModel.load(). The smart reply feature isn't even used on every app launch. How do you eliminate the 3-second penalty without removing the feature?"
Common Mistake: "Load the model in a background thread at launch." This still consumes 3 seconds of CPU/memory resources during launch, competing with UI rendering and causing janky animations even if the main thread isn't blocked.
Realistic Solution: The 3-second MLModel.load() includes: model file mmap (~50ms), CoreML compilation/cache check (~500ms), ANE program generation (~1500ms), and activation buffer allocation (~950ms). This is unavoidable for the first load but can be deferred and optimized:
(1) Lazy loading — don't load the model at app launch. Load it when the user first opens a conversation (where smart reply is relevant). The 3-second delay happens once, in context, and can be hidden behind a "Preparing smart replies..." shimmer animation. Subsequent loads use the cached compilation: ~200ms.
(2) Pre-compilation at install time — use CoreML's MLModel.compileModel(at:) during the app's first launch or after an update. This front-loads the 1.5-second compilation step. Store the compiled .mlmodelc in the app's cache directory. Subsequent loads skip compilation: 3s → 1.5s.
(3) Warm-up during idle — after the app launches and the UI is fully rendered, schedule a low-priority background task (using DispatchQueue.global(qos: .utility)) that loads the model during idle time. If the user opens a conversation within 5 seconds, the model may already be loaded. If not, the load completes invisibly.
(4) Model partitioning — split the 1.5B model into an embedding layer (50 MB, loads in 200ms) and the transformer blocks (700 MB, loads in 2.8s). Load the embedding layer at launch (200ms overhead — acceptable). Load the transformer blocks lazily. The embedding layer enables instant query encoding; the full model loads in the background for generation.
(5) Persistent background process — on iOS, use BGAppRefreshTask to keep the model warm in memory. If the app was recently used, iOS may keep it in memory (not terminated), and the model is already loaded on next launch. This works for ~60% of launches (when the app wasn't evicted).
Napkin Math: Current launch: 1.5s (app) + 3s (model) = 4.5s. Lazy load: 1.5s (app) + 0s (model deferred) = 1.5s launch. First smart reply: +3s (model load) = 3s wait. With pre-compilation: first smart reply = 1.5s. With idle warm-up (5s after launch): if user opens conversation after 5s, model already loaded → 0ms wait. Usage data shows 80% of users open a conversation >10s after launch → 80% of users never see the model load delay. The 20% who open immediately see a 1.5-3s shimmer — acceptable.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "You're running federated learning across 10,000 Android devices to train a next-word prediction model. Each round, the server selects 100 devices to train locally for 5 epochs and upload gradients. The round consistently fails — only 30-40 devices complete in the 10-minute window. Your fleet includes: Snapdragon 8 Gen 3 flagships (20%), Snapdragon 7 Gen 1 mid-range (35%), MediaTek Dimensity 700 budget (30%), and Samsung Exynos 1380 budget (15%). Why does device heterogeneity break federated learning, and how do you fix the round completion rate?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Select only flagship devices for training — they're fast enough." This introduces selection bias: flagship users have different typing patterns (more affluent, different demographics) than budget phone users. The model learns a biased distribution.
Realistic Solution: The round fails because of the straggler problem — the round completes only when all 100 selected devices finish. The slowest device determines round latency:
Training speed by device tier:
With random selection of 100 devices: expected 45 budget devices (30% Dimensity + 15% Exynos). All 45 will timeout. Round completion: 55/100 = 55%. But the server requires 80% completion for a valid round → round fails.
Fixes:
(1) Tiered training — assign different workloads by device capability. Flagships: 5 epochs, batch size 32. Mid-range: 3 epochs, batch size 16. Budget: 1 epoch, batch size 8. Equalize wall-clock time to ~5 minutes across all tiers. Weight the gradient contributions by number of examples processed.
(2) Asynchronous aggregation — don't wait for all devices. Aggregate gradients as they arrive. After 7 minutes, aggregate whatever is available (typically 70-80 devices). Use FedBuff (buffered async FL): aggregate every 50 gradient arrivals, regardless of which devices they came from.
(3) Over-selection — select 150 devices, accept the first 100 to complete. This naturally selects faster devices per round while maintaining fleet diversity over many rounds (budget devices complete in some rounds when they happen to be idle).
(4) Gradient compression — reduce upload size from 200 MB (full FP32 gradients) to 2 MB (top-1% sparse + quantized). Upload time on cellular: 200 MB / 5 Mbps = 320 seconds → 2 MB / 5 Mbps = 3.2 seconds. For budget devices on slow networks, upload time was the actual bottleneck, not training.
Napkin Math: Round budget: 10 minutes. Training time: flagship 1.7 min, mid-range 6.7 min, budget 13.3 min. Upload (200 MB, 5 Mbps): 5.3 min. Total: flagship 7 min ✓, mid-range 12 min ✗, budget 18.6 min ✗. With tiered training + gradient compression: flagship 1.7 + 0.05 = 1.75 min. Mid-range: 4 + 0.05 = 4.05 min. Budget: 2.7 + 0.05 = 2.75 min. All under 10 minutes. Round completion: 95%+. Over-selection (150 → 100): expected wait for 100th device ≈ P95 of device completion time ≈ 5 min. Rounds complete reliably.
📖 Deep Dive: Volume I: Data Engineering
</details>Interviewer: "Your PM wants on-device image generation (Stable Diffusion-style) on an iPhone 16 Pro (A18 Pro, 8 GB RAM). The model: a 1B parameter U-Net (INT8 = 1 GB weights), CLIP text encoder (340M params, FP16 = 680 MB), and VAE decoder (80M params, FP16 = 160 MB). Total weights: 1.84 GB. It fits in the ~5 GB available memory. But during the 20-step denoising loop, the app gets jetsammed at step 12. The weights haven't changed. What's consuming memory during denoising?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model is too big — quantize more aggressively." The weights fit. The problem is the activation memory during the U-Net's forward pass, which peaks during the denoising loop.
Realistic Solution: Stable Diffusion's U-Net has a unique memory profile due to its architecture:
(1) U-Net skip connections — the encoder downsamples through 4 resolution levels (64→32→16→8), storing feature maps at each level for the decoder's skip connections. At 512×512 output (64×64 latent): Level 1: 64×64×320 × 2 bytes = 2.5 MB. Level 2: 32×32×640 = 1.25 MB. Level 3: 16×16×1280 = 0.31 MB. Level 4: 8×8×1280 = 0.08 MB. Total skip connections: ~4.1 MB per step. Manageable.
(2) Self-attention memory explosion — the U-Net has self-attention at 32×32 and 16×16 resolutions. At 32×32: sequence length = 1024 tokens. Attention matrix: 1024 × 1024 × 8 heads × 2 bytes = 16 MB per attention layer. With 6 attention layers at this resolution: 96 MB. At 16×16: 256 tokens, much smaller.
(3) Cross-attention with CLIP — the text conditioning uses cross-attention. Key/value from CLIP (77 tokens × 768 dim) × query from U-Net (1024 tokens at 32×32). Cross-attention matrix: 1024 × 77 × 8 heads × 2 bytes = 1.2 MB per layer. Small individually, but 12 cross-attention layers = 14.4 MB.
(4) The real killer: intermediate activations during backprop-free inference — CoreML's memory planner must keep all intermediate tensors alive that are needed by downstream ops. The U-Net's skip connections mean encoder activations must survive until the decoder uses them. Peak activation memory: ~800 MB at the U-Net's bottleneck (all encoder activations + current decoder activations + attention matrices).
(5) Cumulative per-step overhead — each denoising step allocates temporary buffers. If CoreML doesn't perfectly reuse buffers between steps (a known issue with some model structures), memory grows ~50 MB per step. After 12 steps: 600 MB of leaked temporary buffers.
Total at step 12: 1.84 GB (weights) + 800 MB (peak activations) + 600 MB (buffer leak) + 500 MB (app + OS overhead) = 3.74 GB. Jetsam limit on iPhone 16 Pro: ~4 GB. Step 12 pushes past the limit.
Fix: (1) Sequential U-Net execution — run encoder, save skip features to disk, free encoder memory, then run decoder. Trades 200ms I/O per step for ~400 MB memory savings. (2) Attention slicing — compute attention in chunks of 256 tokens instead of all 1024 at once. Peak attention memory: 96 MB → 24 MB. (3) Explicit buffer reuse — use CoreML's MLPredictionOptions with pre-allocated output buffers to prevent per-step allocation growth. (4) Generate at 256×256 — activations scale quadratically with resolution: 800 MB → 200 MB.
Napkin Math: Weights: 1.84 GB. Peak activations (512×512): 800 MB. Buffer leak (12 steps × 50 MB): 600 MB. App overhead: 500 MB. Total: 3.74 GB > 4 GB jetsam limit at step 12. With attention slicing + buffer reuse: 1.84 + 0.53 + 0 + 0.5 = 2.87 GB. Headroom: 1.13 GB. Completes all 20 steps. With 256×256: 1.84 + 0.2 + 0 + 0.5 = 2.54 GB. Ample headroom. Generation time: 20 steps × 1.5s/step = 30 seconds at 512×512. Acceptable for a "generate" button UX.
📖 Deep Dive: Volume I: HW Acceleration
</details>Interviewer: "Your news app uses an on-device ML model (small transformer, 10M params, INT8) on a Snapdragon 8 Elite to predict which articles a user will click, and sends push notifications for high-confidence predictions. The model has 78% precision — seemingly good. But after launch, the app's rating drops from 4.5 to 2.1 stars, with reviews saying 'too many irrelevant notifications.' The model is working as designed. What went wrong?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "78% precision means 78% of notifications are relevant — that's good enough." This ignores the base rate problem and the asymmetric cost of false positives in notifications.
Realistic Solution: The model is technically accurate but the system design is fundamentally flawed:
(1) Base rate problem — the app publishes 500 articles/day. The user clicks ~10 articles/day (2% click rate). The model scores all 500 articles and sends notifications for those above the confidence threshold. At 78% precision and high recall: the model flags ~50 articles as "will click." 78% of 50 = 39 correct predictions. But the user only wanted 10 articles — they receive 50 notifications for 10 desired articles. The 11 false positives (22%) feel like spam, and even the 39 true positives are excessive (the user didn't want to be notified about all of them).
(2) Notification fatigue asymmetry — a missed notification (false negative) has near-zero cost — the user browses the app and finds the article. A false positive notification interrupts the user's day, makes their phone buzz, and erodes trust. The cost ratio is ~100:1 (false positive : false negative). At 78% precision, the expected annoyance per day: 11 false positives × 100 cost = 1100 annoyance units. The expected value of true positives: 39 × 1 = 39 utility units. Net: -1061. The feature destroys more value than it creates.
(3) Temporal clustering — the model processes articles in batches (when new articles are published). If 10 articles are published simultaneously, the user might receive 5 notifications in 2 minutes. Even if all 5 are relevant, the burst feels like spam.
Fix: (1) Raise the precision threshold to 95%+ — accept lower recall. Send 5 notifications/day instead of 50. Users prefer missing some articles to being spammed. (2) Rate limiting — max 3 notifications per day, max 1 per hour. Select the top-3 highest-confidence predictions. (3) User feedback loop — track notification dismissals vs opens. If the user dismisses 3 consecutive notifications, halve the notification frequency. (4) Digest mode — instead of per-article notifications, send one daily digest: "5 articles you might like." One notification, zero spam perception.
Napkin Math: 500 articles/day, 2% click rate = 10 relevant. Model at 78% precision, 80% recall: flags 10.3 articles correctly, 2.9 incorrectly → ~13 notifications. User annoyance: 2.9 × 100 = 290 annoyance units. User value: 10.3 × 5 = 51.5 utility units. Net: -238.5 (negative value). At 95% precision, 40% recall: flags 4 correctly, 0.2 incorrectly → ~4 notifications. Annoyance: 0.2 × 100 = 20. Value: 4 × 5 = 20. Net: 0 (break-even). With rate limit of 3/day: annoyance capped at 0.6 × 100 = 60 max. The model's precision must be >97% for notifications to have positive expected value.
📖 Deep Dive: Volume I: Data Engineering
</details>Interviewer: "You train a single PyTorch image classification model and deploy it as CoreML on iOS (iPhone 16 Pro, A18 Pro) and TFLite on Android (Galaxy S24, Snapdragon 8 Gen 3). Both use INT8 quantization. For the same test image, the iOS model outputs 92% confidence for 'golden retriever' while the Android model outputs 71% confidence. The predicted class is the same, but the confidence gap causes your app's UX to behave differently (iOS shows 'Definitely a golden retriever!' while Android shows 'Might be a golden retriever'). Explain the three conversion steps where confidence diverges."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The models are quantized differently — requantize with the same calibration data." Even with identical calibration data, the conversion pipelines produce different quantized models because they make different algorithmic choices.
Realistic Solution: Three conversion steps introduce divergence:
(1) Graph optimization differences — coremltools and TFLite's converter apply different graph transformations before quantization. CoreML fuses Conv+BN+ReLU into a single op, then quantizes the fused op. TFLite fuses Conv+BN but keeps ReLU separate, quantizing each independently. The fused quantization has a different effective scale factor than the separate quantization, because the fused op's output range differs from the unfused chain's intermediate ranges. This shifts intermediate activations by 1-3%.
(2) Calibration quantization algorithm — CoreML uses a min-max calibration by default (maps the observed min/max of activations to the INT8 range). TFLite uses a percentile-based calibration (clips the top/bottom 0.1% of activation values to reduce the impact of outliers). If a calibration image produces an outlier activation of 50.0 while typical activations are 0-5.0: CoreML maps [0, 50] to INT8, giving only 25 levels for the [0, 5] range (where 99% of values live). TFLite clips to [0, 5.2] and maps that to INT8, giving 245 levels for the useful range. The TFLite model has 10× better effective precision for typical activations.
(3) Softmax implementation — the final softmax converts logits to probabilities. CoreML on the ANE computes softmax in FP16: exp() and division in 16-bit. TFLite on the Hexagon NPU uses a lookup-table approximation for exp() in INT8, then converts to FP32 for the division. The LUT approximation compresses the dynamic range of the logits. If the true logits are [4.2, 2.1, 0.5, ...], the FP16 softmax produces [0.92, 0.06, 0.02]. The INT8 LUT softmax produces [0.71, 0.18, 0.11] — the LUT's coarser exp() approximation flattens the probability distribution.
Fix: (1) Export the model to ONNX first, then convert to both CoreML and TFLite from the same ONNX graph — this standardizes graph optimizations. (2) Use the same calibration algorithm (force both to use min-max or both to use percentile). (3) Run softmax in FP32 on CPU for both platforms — costs ~0.05ms but guarantees identical confidence scores.
Napkin Math: Logits (FP32 reference): [4.2, 2.1, 0.5, -1.3]. FP32 softmax: [0.917, 0.056, 0.023, 0.004]. CoreML FP16 softmax: exp(4.2) = 66.69 (FP16: 66.75, 0.09% error). Result: [0.919, 0.055, 0.022, 0.004]. TFLite INT8 LUT: exp(4.2) ≈ 64 (nearest LUT entry). exp(2.1) ≈ 8. Softmax: 64/(64+8+2+0.3) = 0.861. Wait — that gives 86%, not 71%. The additional divergence comes from the quantized logits themselves being different (step 1 and 2): TFLite logits might be [3.8, 2.3, 0.8, -1.0] after different quantization. Softmax of [3.8, 2.3, 0.8, -1.0] in FP32 = [0.71, 0.16, 0.04, 0.006]. The logit quantization error (±0.4) dominates the softmax implementation error (±0.02).
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "Your new object detection model is running on a high-end Android phone's NPU. The NPU spec sheet boasts 10 TOPS (INT8), yet your model's actual performance is only 2 TOPS equivalent, with a significant portion of the NPU staying idle. What's the most likely bottleneck, and how would you diagnose it?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The NPU drivers are buggy," or "It's thermal throttling." While these are possibilities, the NPU being largely idle points to a different fundamental limitation.
Realistic Solution: The most likely bottleneck is memory bandwidth. The NPU can perform calculations very quickly, but it's starved for data because the memory subsystem (DRAM) cannot feed it fast enough. This often happens with models that have a low "compute intensity" (FLOPs per byte of data accessed) or those with large intermediate tensors that frequently need to be moved between memory and the NPU. The NPU spends significant cycles waiting for data to be fetched from DRAM.
adb shell dumpsys meminfo / adb shell top: Monitor overall memory pressure and active processes.Napkin Math: A 10 TOPS (INT8) NPU processes 10 * 10^12 operations/second. If each INT8 operation requires, on average, 0.5 bytes of input and 0.5 bytes of output (a simplified model for compute-bound operations), that's roughly 10 TB/s of data movement required to sustain peak. A typical LPDDR5 mobile memory controller offers ~50 GB/s peak bandwidth. If the model's actual data movement requirement exceeds this, or if the effective bandwidth is lower due to contention or access patterns, the NPU will be starved. For example, if a model truly needs 25 GB/s of data, the effective TOPS will be limited to 2.5 TOPS (25 GB/s / 10 TB/s * 10 TOPS = 2.5 TOPS).
Key Equation:
Effective TOPS ≈ (Memory Bandwidth / Data per Op) * Compute Intensity(simplified for illustrative purposes)
📖 Deep Dive: Volume I: Chapter 6 - Performance Bottlenecks
</details>Interviewer: "Your team is deploying a new semantic segmentation model to millions of mobile devices. You need to achieve sub-50ms latency and a model size under 20MB. You're considering dynamic range quantization (DRQ) vs. quantization-aware training (QAT). Describe the trade-offs in terms of model development, deployment, performance, and memory footprint on device, especially considering future over-the-air (OTA) model updates."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "QAT is always superior for accuracy and performance, so we should always use it." This overlooks practical deployment and development costs, especially for continuous updates.
Realistic Solution: The choice between DRQ and QAT involves significant trade-offs:
Dynamic Range Quantization (DRQ / Post-Training Dynamic):
Quantization-Aware Training (QAT):
Conclusion for the scenario: Given the tight latency (sub-50ms) and size (under 20MB) constraints, QAT is likely required to achieve optimal performance and accuracy. However, the team must acknowledge the increased development overhead and bake a robust QAT pipeline into their OTA update strategy. If the model evolves frequently, the QAT overhead per update might be a deterrent.
Napkin Math: A 100MB FP32 model (4 bytes/parameter) can be reduced to 25MB (1 byte/parameter) with INT8 QAT for weights and activations, easily fitting the 20MB target. With DRQ, weights are 25MB, but if activations need to be dynamically converted, a single large activation tensor of 100x100x128 elements (1.28M elements) would require ~1.28M operations for conversion from FP32 to INT8, adding compute and memory bandwidth overhead at runtime.
Key Equation:
Model Size (INT8) = Model Size (FP32) / 4(for weights/biases)
📖 Deep Dive: Volume I: Chapter 7 - Quantization
</details>Interviewer: "Your mobile ML model performs exceptionally well for the first 10-15 seconds after launch, achieving 30 FPS. However, after this initial period, the frame rate consistently drops to 15-20 FPS and stays there. The phone isn't running any other heavy applications. What's the primary cause for this behavior, and how would you confirm it?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "There's a memory leak causing garbage collection pauses," or "The NPU is being deallocated after a grace period."
Realistic Solution: This is a classic symptom of thermal throttling. Modern mobile SoCs (System-on-Chips) are designed to burst to peak performance for short durations, but sustained high compute loads generate significant heat. Once the internal temperature sensors detect a pre-defined threshold has been breached, the SoC's firmware will automatically reduce clock frequencies (CPU, GPU, NPU) and/or voltage to prevent overheating, ensuring device longevity and user comfort. This reduction in clock speed directly leads to a sustained drop in performance.
adb shell dumpsys cpuinfo, adb shell dumpsys gfxinfo, or vendor-specific profiling tools (e.g., Qualcomm Snapdragon Profiler) to monitor actual clock speeds over time. You should observe a drop after the initial burst.adb shell dumpsys battery (shows battery temperature) or adb shell cat /sys/class/thermal/thermal_zone*/temp (shows various sensor temperatures) to observe temperature changes correlating with the performance drop.Napkin Math: A typical mobile SoC might have a peak power consumption of 8-10W for a few seconds, but a sustained thermal design power (TDP) of only 3-5W for ML inference. If your model initially draws 8W, it can only sustain that for a short period before hitting thermal limits. Dropping from 30 FPS to 15-20 FPS implies a 33-50% reduction in effective compute, which corresponds to the SoC reducing clock speeds to stay within the sustainable 3-5W range.
Key Equation:
Power ≈ C * V^2 * f(where C is capacitance, V is voltage, f is frequency). Reducingf(frequency) orV(voltage) significantly reduces power consumption and thus heat.
📖 Deep Dive: Volume I: Chapter 5 - Power & Energy
</details>Interviewer: "You're integrating a novel graph neural network (GNN) model on a specific Android device with a proprietary NPU (e.g., a custom Google Tensor NPU or Qualcomm Hexagon DSP). The GNN has a custom aggregation operator not natively supported by TensorFlow Lite or the vendor's NPU SDK. Running this operator on the CPU causes a 20ms latency spike, making the overall model too slow. How would you approach accelerating this custom operator?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just implement it as a TFLite custom op, which will run on the CPU." While this addresses the API integration, it fails to solve the critical performance issue.
Realistic Solution: Accelerating a critical custom operator requires deep integration with the vendor's hardware acceleration stack and often involves low-level engineering:
Napkin Math: If the CPU execution of the custom op takes 20ms, and the target is <5ms, a 4-5x speedup is needed. A typical NPU/DSP can offer 10-100x speedup for suitable operations compared to a single CPU core, making custom kernel development a viable path if the operation maps well to the accelerator's architecture (e.g., matrix multiplications, element-wise operations, reductions).
Key Equation:
Latency_total = Sum(Latency_op_i)whereLatency_op_i = Max(Compute_time_i, Memory_access_time_i, Transfer_time_i)
📖 Deep Dive: Volume I: Chapter 6 - Hardware Accelerators
</details>Interviewer: "Your mobile app is experiencing intermittent Out-Of-Memory (OOM) crashes on certain Android devices, specifically when loading a 100MB ML model. However, adb shell dumpsys meminfo shows that the device still has 500MB of free physical RAM. Why might the app be crashing with OOM despite seemingly ample free memory?"
Common Mistake: "The model is simply too large for the device's total RAM." This doesn't explain why it crashes with 500MB of reported free memory.
Realistic Solution: The OOM crash, despite reported free physical RAM, points to issues with virtual memory address space fragmentation or process-specific memory limits, rather than a lack of total physical memory.
Napkin Math: If a 32-bit process has a 2GB virtual address space and 1GB is already allocated in scattered 1MB chunks, requesting a 100MB contiguous block will fail even though 1GB of virtual memory is technically "free." For 64-bit systems, virtual address space fragmentation is less of an issue, but the other points (heap limits, LMK) still apply.
Key Equation:
Available Contiguous Virtual Memory < Requested Allocation Size
📖 Deep Dive: Volume I: Chapter 5 - Memory Management
</details>Interviewer: "You have a complex ML model with a mix of convolutional, recurrent, and custom attention layers. Profiling shows that convolutions run best on the NPU, recurrent layers on the DSP, and custom attention layers (due to their sparsity patterns) are most efficient on the CPU. How would you design the execution strategy to minimize overall latency and power consumption on a mobile SoC with these heterogeneous compute units?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just use a TFLite delegate and let it figure it out." While delegates provide a basic abstraction, they might not make optimal, fine-grained decisions for complex heterogeneous graphs, especially regarding cross-device data transfers and pipelining.
Realistic Solution: This requires fine-grained heterogeneous task graph partitioning and scheduling, aiming to balance computation across devices, minimize data movement, and exploit parallelism.
Napkin Math: If an NPU-CPU data transfer costs 5ms for a specific tensor, and the CPU operation itself takes 2ms, it's better to keep that operation on the NPU if it takes <7ms there, even if the CPU is slightly faster for the op itself. The cost function for partitioning needs to consider
(compute_cost_on_device + transfer_cost_to_device).
Key Equation: For pipelined execution,
Latency_total = Max(Compute_path_1, Compute_path_2, ..., Compute_path_N) + Pipeline_setup_overhead. For sequential,Latency_total = Sum(Compute_cost_i + Transfer_cost_i).
📖 Deep Dive: Volume I: Chapter 6 - Heterogeneous Computing
</details>Interviewer: "You've deployed a tiny image preprocessing model (only 500KB, 10M FLOPs) on a mobile device, running entirely on the CPU. To your surprise, its latency is consistently 15ms, which is much higher than expected for such a small model. What's a common, often overlooked factor that could be causing this high latency on the CPU, and how would you investigate?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The CPU is just too slow for ML," or "There's too much overhead from the ML framework." While possible, for such a small model, these often aren't the primary causes.
Realistic Solution: A common culprit for small, CPU-bound models with unexpectedly high latency is poor cache locality and inefficient memory access patterns.
perf on Linux or Android Studio's CPU profiler. Look for high cache miss rates (L1, L2, L3) and high memory stall cycles. These indicate the CPU is spending a lot of time waiting for data from memory.Napkin Math: A typical L1 cache hit takes ~1-4 cycles, L2 ~10-20 cycles, L3 ~50-100 cycles, and DRAM ~200-400 cycles. If a small model with 10M FLOPs (e.g., 10 million operations) has a 10% L1 miss rate and a 5% L2 miss rate, the accumulated memory access penalty can easily dominate the actual compute time. For instance, 1M L1 misses (at 10 cycles each) = 10M cycles, and 0.5M L2 misses (at 50 cycles each) = 25M cycles. At a 2GHz CPU, this alone adds (10M+25M) cycles / 2 * 10^9 cycles/sec = 17.5ms, turning a potentially few ms compute into 15ms+.
Key Equation:
Execution Time = (Compute Cycles + Memory Stall Cycles) / Clock Frequency
📖 Deep Dive: Volume I: Chapter 5 - CPU Architecture
</details>Interviewer: "Your team wants to push the envelope on model size and latency, aiming for INT4 quantization for a critical on-device generative AI model. What are the significant technical challenges you anticipate with INT4 quantization compared to INT8, both in terms of hardware support and accuracy preservation, and how would you mitigate them?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "It's just halving the bit-width; calibration will fix it." This vastly oversimplifies the severe challenges introduced by INT4.
Realistic Solution: INT4 quantization introduces significantly more acute challenges than INT8, pushing the limits of current hardware and software techniques:
Challenges:
Mitigation Strategies:
Napkin Math: An INT8 tensor has a quantization step size of
(max - min) / 255. An INT4 tensor has(max - min) / 15. For the same dynamic range, the INT4 step size is ~17x larger. The quantization noise is proportional to the square of the step size, meaning INT4 noise can be ~289x higher than INT8 for the same range, if not managed carefully.
Key Equation:
Quantization Error ∝ (Range / (2^bits - 1))^2
📖 Deep Dive: Volume I: Chapter 7 - Advanced Quantization
</details>Interviewer: "You're tasked with deploying a computer vision model on both Android and iOS, with a strict power consumption budget for inference (e.g., <100mW average over 30 seconds). You've benchmarked the model using TensorFlow Lite (TFLite) on Android and Core ML on iOS. You observe that while TFLite on Android achieves the latency target, its power draw is consistently 20-30% higher than Core ML on iOS for a similar model and device class. What factors could explain this power discrepancy, and how would you optimize TFLite's power efficiency?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Android devices are just less power-efficient than iOS." While general hardware differences exist, this doesn't explain a framework-specific discrepancy for a similar model and device class.
Realistic Solution: The power discrepancy often stems from differences in how the ML frameworks interact with the underlying hardware, operating system, and their respective drivers/delegates.
Factors Explaining the Discrepancy:
Optimization for TFLite's Power Efficiency:
TFLiteInterpreter::GetLastNnapiErrorMessage(), adb logcat) and vendor profilers to verify that all critical operations are being delegated to the NPU. Address any fallback to CPU.Napkin Math: If model inference takes 50ms, and the NPU consumes 200mW while the CPU consumes 500mW. A 10% CPU fallback (5ms of CPU time per inference) adds 5ms * 500mW = 2.5mJ of energy, compared to NPU's 5ms * 200mW = 1mJ. Over 30 seconds of continuous inference (600 inferences), this difference accumulates significantly.
Key Equation:
Total Energy = Sum(Power_device_i * Time_device_i)
📖 Deep Dive: Volume I: Chapter 5 - Power & Energy
</details>Interviewer: "Your real-time ML model on Android generally achieves a 30ms latency, but occasionally you observe unpredictable spikes up to 100-200ms. These spikes don't correlate with thermal throttling, NPU utilization drops, or obvious background app activity. How would you diagnose and mitigate these latency spikes, focusing on memory management within the ML inference pipeline?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "It's OS scheduler contention or garbage collection." While these are general causes for latency spikes, the problem statement rules out obvious background activity and stable NPU utilization, suggesting a more specific memory management issue within the ML pipeline itself.
Realistic Solution: The unpredictable latency spikes, when other common causes are ruled out, often point to dynamic memory allocation overhead and heap fragmentation within the inference process.
Diagnosis:
malloc/free calls can be expensive. They might contend for locks on the heap, trigger searches for free blocks, or even initiate OS-level page allocations.adb shell am dumpheap, or integrate custom memory allocators like jemalloc or tcmalloc to analyze native heap behavior, identify frequent allocations, and detect fragmentation.Mitigation:
malloc/free calls during the critical path, significantly reducing overhead and fragmentation. The pool is reset or managed between inferences.ByteBuffer instances or native pointers.Napkin Math: A single
malloccall can take anywhere from tens of nanoseconds to several microseconds depending on the allocator, heap state, and contention. If an inference involves hundreds or thousands of such calls, this overhead accumulates. 1000malloccalls at an average of 50µs each (due to fragmentation/contention) would add 50ms of overhead, easily explaining a jump from 30ms to 80ms+.
Key Equation:
Latency_spike = Sum(Malloc_overhead_i + Free_overhead_i)
📖 Deep Dive: Volume I: Chapter 5 - Memory Management
</details>Interviewer: "You've successfully quantized a large image segmentation model from FP32 to INT8, reducing its size from 200MB to 50MB. While the model load time improved, the inference latency only decreased by 15%, not the 4x you expected from the size reduction. What's a primary reason for this limited latency improvement, especially on a memory-bound mobile NPU?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The NPU isn't truly 4x faster for INT8." While true that INT8 ops might not be 4x faster than FP32 ops on all NPUs, the question hints at a memory-bound scenario, where computation speed isn't the primary bottleneck.
Realistic Solution: The primary reason is that memory bandwidth often becomes the bottleneck before computation throughput is fully utilized, especially for models with many layers or large intermediate tensors. While the model weights are smaller, the activations (intermediate tensors) still need to be moved between on-chip caches, main memory (DRAM), and the NPU's processing units. If the model is already memory-bound in FP32, reducing weight size alone won't significantly improve performance if the activation data movement remains the dominant factor. Furthermore, the NPU might have internal data paths that are still 32-bit wide for intermediate results, or data transfer overheads (e.g., PCIe/AXI bus) might dominate.
Napkin Math: Assume a memory-bound layer needs to read
Wbytes of weights andAbytes of activations per inference. FP32: Total memory access $\approx W_{FP32} + A_{FP32}$. INT8: Total memory access $\approx W_{INT8} + A_{INT8}$. If $W_{FP32} = 100MB$, $A_{FP32} = 50MB$ (per inference), and $W_{INT8} = 25MB$, $A_{INT8} = 50MB$. FP32 access: $150MB$. INT8 access: $75MB$. This should be 2x faster in a perfectly memory-bound scenario. However, if the NPU has a peak compute of 10 TOPS (INT8) but only 25GB/s memory bandwidth, and the layer requires 100GB/s of memory bandwidth at peak compute, then memory bandwidth is the limiter. Even if data access is halved, if it's still 50GB/s, it's 2x the available bandwidth, so it's still bottlenecked. The 15% improvement might come from some layers becoming compute-bound or minor cache benefits.
Key Equation: $Latency \approx \max(\frac{Memory_Access_Volume}{Memory_Bandwidth}, \frac{FLOPs}{Compute_Throughput})$
📖 Deep Dive: Volume I: Chapter 6 - Hardware for Deep Learning
</details>Interviewer: "You have a very small, simple ML model – say, a single 100-neuron dense layer. When you deploy it on a modern mobile SoC, you observe that running it on the CPU often yields lower latency than trying to offload it to the dedicated NPU. Why would this be the case?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The NPU isn't optimized for small models." While partially true, it doesn't explain why it's slower. The core issue is the overhead.
Realistic Solution: Dedicated NPUs often have significant startup and data transfer overheads that overshadow the benefits of their higher theoretical throughput for very small workloads.
Napkin Math: CPU execution time: $T_{CPU_compute} \approx 50 \mu s$ (for a small dense layer). NPU execution time: $T_{NPU_startup} + T_{data_transfer} + T_{NPU_compute}$. $T_{NPU_startup} \approx 100 \mu s$ (driver, power-up). $T_{data_transfer} \approx 20 \mu s$ (e.g., 1MB data @ 50MB/s effective transfer rate). $T_{NPU_compute} \approx 10 \mu s$ (if NPU is truly faster). Total NPU: $100 + 20 + 10 = 130 \mu s$, which is much higher than $50 \mu s$ on CPU.
Key Equation: $Total_Latency = Fixed_Overhead + \frac{Workload}{Throughput}$
📖 Deep Dive: Volume I: Chapter 6 - Hardware for Deep Learning
</details>Interviewer: "You're deploying a real-time audio processing model on a mobile device. The input audio amplitude can vary wildly depending on the environment (quiet room vs. loud concert). Static INT8 post-training quantization (PTQ) leads to significant accuracy degradation during high-amplitude spikes or very low-amplitude signals. How would you adapt your quantization strategy to maintain accuracy while still leveraging INT8 performance on mobile hardware?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Re-quantize the model on the fly." While conceptually related, re-quantizing the entire model's weights dynamically is too slow and memory-intensive for real-time mobile inference.
Realistic Solution: The most effective approach for varying input distributions while maintaining INT8 performance is dynamic range quantization for activations, combined with static quantization for weights.
quantize_and_dequantize operations inserted into the graph, where the quantization parameters are determined per-tensor.Napkin Math: Dynamic Quantization: For each activation tensor of size
Nelements, calculating min/max takesO(N)operations. This overhead is typically much smaller than the actual matrix multiplication/convolution operations, especially for larger tensors. E.g., for a 1MB activation tensor (250k FP32 elements), finding min/max takes ~0.25 MFLOPS. A typical NPU can do GigaFLOPS, so this overhead is negligible. If a model has 100 layers, and each activation requires dynamic quantization, total overhead for min/max calculation is $100 \times O(N_{avg})$.
Key Equation: $q = round(x / S) + Z$, where $S = (max - min) / (2^B - 1)$ and $Z = -min / S$. (S and Z are dynamically computed for activations).
📖 Deep Dive: Volume I: Chapter 7 - Quantization
</details>Interviewer: "Your mobile application intermittently crashes with an Out-Of-Memory (OOM) error specifically when trying to load a 500MB ML model, even though the device reports several gigabytes of free RAM. The crash is not consistent; it happens more frequently on older devices or after the app has been running for a while. What's the most likely root cause, and how would you debug and mitigate it?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The device doesn't have enough RAM." If the device reports GBs of free RAM, total RAM isn't the issue. It's about how that RAM is available.
Realistic Solution: The most likely root cause is memory fragmentation, specifically virtual memory fragmentation or heap fragmentation within the app's process. Even if there's enough total free RAM, the operating system or the application's memory allocator might not be able to find a contiguous block of virtual memory large enough to satisfy the 500MB allocation request.
mmap a model file, or a single large malloc), the OS might struggle to find a virtual address range large enough to back it, even if physical pages are available.Debugging & Mitigation:
adb shell dumpsys meminfo <package_name> to inspect process memory. On Linux (Android kernel), /proc/<pid>/maps can show virtual memory layout.mmap): If the model is a file, mmap it instead of reading it into a dynamically allocated buffer. This allows the OS to handle paging and potentially load parts of the model on demand, reducing the need for a single contiguous block in virtual memory (though the file itself needs to be contiguous on disk). It also bypasses heap fragmentation.malloc/free calls and reduces heap fragmentation for frequently used ML tensors.largeHeap flag: For very memory-intensive apps, setting android:largeHeap="true" in the manifest can give the app a larger heap size, but it's a palliative, not a cure for fragmentation.Napkin Math: A 500MB allocation on a heap with 2GB total used memory. If the average free block size is 1MB, and the largest contiguous free block is 100MB, the 500MB allocation will fail, even if 1GB of total free memory exists.
Key Equation: $Total_Free_Memory \ne Largest_Contiguous_Free_Memory$
📖 Deep Dive: Volume I: Chapter 5 - Operating Systems
</details>Interviewer: "You are designing an on-device ML pipeline for an augmented reality application. The pipeline involves pre-processing (CPU), a light-weight pose estimation model (NPU-preferred), and a heavy visual effects rendering step (GPU). Your goal is to achieve minimal end-to-end latency while adhering to a strict 2W average power budget for the entire pipeline, considering the mobile SoC's heterogeneous architecture. How would you approach scheduling and resource allocation across these different compute units?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Run everything in parallel on its preferred hardware." While parallelism is good, blindly doing so ignores power constraints and inter-unit communication overheads, potentially leading to thermal throttling or exceeding the power budget.
Realistic Solution: This requires a sophisticated heterogeneous task scheduler aware of device-specific power profiles, latency characteristics, and inter-processor communication (IPC) costs.
Napkin Math: CPU stage: 5ms, 500mW. NPU stage: 10ms, 800mW. GPU stage: 15ms, 1200mW. If run sequentially: Total Latency = 30ms. Total Power (average over 30ms) = (5500 + 10800 + 15*1200) / 30 = (2500 + 8000 + 18000) / 30 = 28500 / 30 = 950mW (well within 2W). If NPU and GPU can run partially in parallel after CPU finish, but GPU needs NPU output: CPU (5ms) -> NPU (10ms) -> GPU (15ms). Total latency 30ms. If power budget is hit, e.g., GPU runs at 1.5W instead of 1.2W for 15ms, average power over 30ms is higher. If GPU is downclocked to 1.0W, its latency might increase to 20ms, increasing total latency to 35ms. The scheduler must find the optimal operating points (frequency/voltage) for each unit to meet both latency and power targets.
Key Equation: $Power \approx C \cdot V^2 \cdot f$, where C is capacitance, V is voltage, f is frequency.
📖 Deep Dive: Volume I: Chapter 6 - Hardware for Deep Learning
</details>Interviewer: "Your mobile application performs real-time video analysis using two ML models concurrently. One model processes full-resolution camera frames (Model A), while another analyzes metadata from a separate sensor stream (Model B). Both models, along with the UI rendering thread, are contending for access to the shared DRAM. You observe performance degradation in both ML inference and UI responsiveness. How would you diagnose and mitigate this memory bandwidth contention issue?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just optimize the models' FLOPs." While reducing FLOPs can help, if the bottleneck is memory bandwidth, simply making compute faster won't solve the contention for data access.
Realistic Solution: This is a classic shared memory bandwidth contention problem.
AHardwareBuffer or ION buffers) instead of memcpy.Napkin Math: A typical LPDDR5 mobile DRAM can offer 50-60 GB/s peak bandwidth. Model A (video): 1080p frame (3MB FP32) @ 30fps = 90MB/s input. If it reads 2x its input size in weights/activations, it's 270MB/s. Model B (metadata): negligible, e.g., 1MB/s. UI rendering: 1080p @ 60fps = 180MB/s (framebuffer updates, textures). Total estimated bandwidth: $270MB/s + 1MB/s + 180MB/s \approx 451MB/s$. This is well within 50GB/s. The issue is often effective bandwidth, cache misses, and bursts. If Model A needs 10GB/s for 10ms, and UI needs 5GB/s for 5ms at the same time, this can cause contention. The "2x its input size" is a simplification; real models can require many multiples of input size in intermediate memory access. A large model might require 100s of MBs/GBs of intermediate tensors to be moved.
Key Equation: $Effective_Bandwidth = \frac{Total_Bytes_Transferred}{Total_Time}$ (often much lower than peak).
📖 Deep Dive: Volume I: Chapter 6 - Hardware for Deep Learning
</details>Interviewer: "You've deployed a high-performance computer vision model on a flagship Android device. Initial tests show excellent latency (e.g., 20ms/frame). However, after about 30-60 seconds of continuous operation, the inference latency consistently jumps to 60-80ms/frame. This happens even with the device plugged into power. What is the most likely cause, and how would you design your ML system to provide more sustained performance?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The device is running out of memory." While memory leaks can degrade performance, a sudden, consistent drop in compute performance after a fixed duration, even on power, strongly points to thermal issues, not memory exhaustion.
Realistic Solution: This is a classic symptom of thermal throttling. Modern mobile SoCs are designed to operate within strict thermal envelopes to prevent overheating and component damage. When a component (CPU, GPU, NPU) generates too much heat over a sustained period, the system's thermal management unit (TMU) reduces its clock frequency and/or voltage (DVFS) to lower power consumption and cool down the chip. This directly impacts performance. Even when plugged in, the device's cooling capacity is limited.
Diagnosis & Design for Sustained Performance:
adb shell dumpsys thermalservice or vendor-specific profilers (e.g., Snapdragon Profiler, ARM Streamline) to monitor core temperatures and clock frequencies of CPU, GPU, and NPU over time. You'll likely see clock frequencies drop significantly when performance degrades.Napkin Math: A typical flagship mobile SoC might have a peak power consumption of 8-10W for short bursts, but a sustained thermal design power (TDP) of only 3-4W for the entire chip. If your ML workload consumes 5W continuously, it will quickly exceed the sustained TDP, leading to throttling. A 20ms inference at 5W means 100mJ per inference. If throttled to 60ms, the power might drop to 1.5W, meaning 90mJ per inference. The system sacrifices speed for thermal stability.
Key Equation: $P_{avg} = \frac{1}{T} \int_0^T P(t) dt \le TDP_{sustained}$
📖 Deep Dive: Volume I: Chapter 6 - Hardware for Deep Learning
</details>Interviewer: "You are tasked with deploying a cutting-edge generative AI model (e.g., a small diffusion model or a compact LLM variant) on a mobile SoC. Achieving near FP32 accuracy is critical, but the model is too large and slow for FP16 inference, and a naive INT8 quantization causes unacceptable accuracy degradation. Some parts of the SoC support INT4, others INT8, and some only FP16/FP32. How would you design a mixed-precision quantization strategy to optimize for this complex trade-off between accuracy, latency, and memory footprint, leveraging the heterogeneous capabilities of the SoC?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Apply INT8 everywhere, then fallback to FP16 for problematic layers." This is a reactive approach. A L6+ candidate should propose a proactive, systematic, and hardware-aware strategy.
Realistic Solution: This requires a highly systematic and hardware-aware approach to mixed-precision quantization, often involving automated tools and iterative refinement.
quantize/dequantize operations and ensuring optimal data transfer.Napkin Math: Model with 100 layers. Each layer can be INT4, INT8, FP16. $3^{100}$ possible configurations (too many). A diffusion model has many convolution/attention blocks. A large MatMul in an attention block might be highly sensitive to INT4, requiring FP16, while subsequent convolutions can be INT8. If a layer is 10x faster in INT4 but drops accuracy by 5%, while another is only 2x faster in INT4 but drops accuracy by 0.1%, the latter is a better candidate for INT4. A key layer might contribute 30% of latency. If FP16 for this layer reduces latency by 5% but INT8 degrades accuracy by 10%, keeping it FP16 is a good trade-off.
Key Equation: $Loss_{mixed} \approx Loss_{FP32} + \sum_{i=1}^{N} \alpha_i \cdot \Delta Loss_i(P_i)$, where $\Delta Loss_i(P_i)$ is accuracy degradation from quantizing layer $i$ to precision $P_i$, and $\alpha_i$ is a sensitivity weight.
📖 Deep Dive: Volume I: Chapter 7 - Quantization
</details>Interviewer: "You are developing a high-throughput, low-latency ML inference engine for an always-on mobile service. You've identified that the standard system malloc/free calls are causing significant performance variability, increased memory fragmentation over long runs, and non-deterministic latency due to OS-level memory management overheads. How would you design and implement a custom memory allocator specifically optimized for the unique characteristics of ML inference workloads on a mobile device?"
Common Mistake: "Just use std::vector and hope for the best." This avoids raw malloc but doesn't solve the underlying issues with the system allocator for high-performance, specialized workloads.
Realistic Solution: A custom memory allocator for ML inference workloads would typically leverage memory pooling or arena allocation strategies, tailored to the predictable nature of tensor allocations.
mmap or a single large malloc). This minimizes OS calls during inference and reduces virtual memory fragmentation.O(1) time without actual deallocations. This completely avoids fragmentation and has minimal overhead.posix_memalign or custom alignment logic within the pool).AHardwareBuffer on Android) to avoid copying data into the custom pool where possible, instead mapping external buffers directly.O(1) allocation/deallocation for most cases.Napkin Math: A typical ML graph might generate 100 intermediate tensors. Each
malloc/freecan take 100-500ns. Total overhead per inference: $200 \times 200ns = 40 \mu s$. With a custom arena allocator, allocation is a pointer increment (e.g., 10ns), and deallocation for the whole inference is 0ns (pointer reset). Total overhead per inference: $100 \times 10ns = 1 \mu s$. This is a significant reduction in overhead. If peak memory for intermediate tensors is 200MB, a custom allocator can ensure this memory is a single contiguous block, avoiding fragmentation issues.
Key Equation: $Total_Memory_Required = Max_{t \in Inference} (\sum_{tensor \in Live_at_t} Size(tensor))$ (Minimized by liveness analysis).
📖 Deep Dive: Volume I: Chapter 5 - Operating Systems
</details>