interviews/cloud/06_advanced_systems.md
This round fills the gaps across compute analysis, memory systems, numerical representation, model architecture costs, power and thermal constraints, and security/fairness — topics that real interviewers test but that were underrepresented in earlier rounds.
➕ Add a Flashcard (Edit in Browser) — see README for the template.
Interviewer: "A junior engineer on your team is frustrated. They just deployed a model on an H100 and the profiler reports only 30% of peak TFLOPS. They want to file a bug against NVIDIA. Before they embarrass themselves, explain why their GPU isn't broken."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The CUDA kernels aren't optimized" or "We need to increase the batch size to saturate the ALUs." Both assume the workload is compute-bound without checking.
Realistic Solution: The GPU is not broken — the workload is memory-bound. The metric that determines whether you hit peak TFLOPS is Arithmetic Intensity: the ratio of compute operations to bytes moved from memory. Every workload lives on a Roofline: below the ridge point, memory bandwidth is the ceiling; above it, compute is the ceiling. If your model's arithmetic intensity is below the GPU's ridge point, no amount of kernel tuning will help — you're waiting on HBM, not ALUs.
Napkin Math: H100 peak = 989 TFLOPS FP16, HBM bandwidth = 3.35 TB/s. Ridge point = $989 \times 10^{12} / 3.35 \times 10^{12}$ = 295 Ops/Byte. If the model's arithmetic intensity is 50 Ops/Byte, the bandwidth ceiling is $50 \times 3.35$ = 167.5 TFLOPS, or $167.5 / 989$ = 17% of peak. The engineer's 30% is actually above the theoretical bandwidth ceiling for their workload — the kernels are fine.
Key Equation: $\text{Attainable FLOPS} = \min\bigl(\text{Peak FLOPS},; \text{BW}_{\text{HBM}} \times I\bigr)$ where $I = \text{FLOPs} / \text{Bytes moved}$
📖 Deep Dive: Volume I: Hardware Acceleration
</details>Interviewer: "We enabled FlashAttention on our Transformer and the profiler shows the same number of FLOPs, but throughput jumped 3×. The team is confused — how can you get faster without doing less math?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "FlashAttention must use some approximation that reduces FLOPs" or "It's just better CUDA kernels." FlashAttention computes mathematically identical attention — no approximation. And the speedup is too large to explain by kernel micro-optimization alone.
Realistic Solution: FlashAttention doesn't reduce FLOPs — it reduces bytes moved. Standard attention materializes the full $N \times N$ attention matrix to HBM: for a 4096-token sequence, that's $4096^2 \times 2$ bytes = 32 MB written and read per head per layer. FlashAttention tiles the computation into SRAM-sized blocks, computing softmax incrementally and never writing the full attention matrix to HBM. The FLOPs are identical, but HBM traffic drops by 4–10×. On the Roofline, this shifts the workload's effective arithmetic intensity from ~50 Ops/Byte to ~200 Ops/Byte, moving it from the memory-bound regime toward the compute-bound regime.
Napkin Math: Standard attention at seq_len=4096, 32 heads, 80 layers: HBM traffic for attention matrices alone = $32 \times 80 \times 4096^2 \times 2 \times 3$ (Q·K, softmax, attn·V) ≈ 258 GB. FlashAttention keeps intermediates in 20 MB of SRAM per SM — HBM traffic drops to ~30 GB. Same FLOPs, ~8× fewer bytes moved. Effective arithmetic intensity jumps from ~50 to ~400 Ops/Byte, crossing the H100's ridge point of 295.
Key Equation: $I_{\text{effective}} = \frac{\text{FLOPs}{\text{attention}}}{\text{Bytes}{\text{HBM}}}$ — FlashAttention holds $\text{FLOPs}$ constant while shrinking $\text{Bytes}_{\text{HBM}}$
📖 Deep Dive: Volume I: Hardware Acceleration
</details>Interviewer: "Your PM asks: 'Can we run Llama-2-7B inference on a single A100 40 GB?' Give me the exact memory breakdown — don't just say 'it fits.'"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "7 billion parameters × 2 bytes = 14 GB, so yes, plenty of room." This accounts for weights only and ignores the two other major memory consumers during inference.
Realistic Solution: You need to budget three components: (1) Model weights — the static cost. (2) KV-cache — grows with batch size and sequence length. (3) Activation memory — temporary buffers for the forward pass. For a single request at 2048 tokens, the budget is tight but feasible on 40 GB. The real question is how many concurrent requests you can serve before the KV-cache pushes you into OOM.
Napkin Math: Llama-2-7B in FP16: Weights = 7B × 2 bytes = 14 GB. KV-cache (32 layers, 32 heads, head_dim=128, seq_len=2048, batch=1) = $2 \times 32 \times 32 \times 128 \times 2048 \times 2$ bytes = 1.07 GB. Activations (peak intermediate buffers) ≈ 0.5 GB. Total ≈ 15.6 GB — fits on A100 40 GB with 24.4 GB free for batching. At batch_size=16: KV-cache balloons to 17.1 GB, total = 31.6 GB — still fits. At batch_size=24: KV-cache = 25.7 GB, total = 40.2 GB — OOM. Scale this to Llama-2-70B at 4K context: a single sequence's KV-cache alone is ~2 GB. 32 concurrent users need 64 GB of KV-cache — nearly all of an H100's 80 GB HBM, leaving almost nothing for the 140 GB of weights (which must be sharded across GPUs).
Key Equation: $\text{VRAM}{\text{inference}} = \underbrace{P \times b_w}{\text{weights}} + \underbrace{2 \times L \times H \times d_h \times S \times B \times b_w}{\text{KV-cache}} + \underbrace{A(B)}{\text{activations}}$
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "We need to train a 70B model on 8× H100 80 GB GPUs. The team enabled ZeRO-3 and claims it'll fit. Walk me through the actual per-GPU memory budget and tell me where it breaks."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "ZeRO-3 shards everything evenly — 70B × 16 bytes / 8 GPUs = 140 GB per GPU. It doesn't fit, so we need more GPUs." This is the right calculation for the sharded state, but it misses that gradient checkpointing changes the equation for activations, which is where the real design decision lives.
Realistic Solution: With ZeRO-3, the model state (weights + gradients + optimizer) is sharded across all 8 GPUs. The mixed-precision training footprint per parameter is 2 (FP16 weights) + 2 (FP16 gradients) + 4 (FP32 master weights) + 8 (FP32 Adam m + v) = 16 bytes. Sharded: 70B × 16 / 8 = 140 GB — exceeds 80 GB. But ZeRO-3 doesn't hold all shards resident simultaneously; during a forward/backward pass, parameters are gathered on-demand and discarded. Peak resident model state is much lower (~20–25 GB). The real memory pressure comes from activations: without checkpointing, storing all intermediate activations for 80 transformer layers at reasonable batch sizes can consume 60+ GB. Gradient checkpointing trades 33% extra compute to store activations only at checkpoint boundaries (every $\sqrt{L}$ ≈ 9 layers), reducing activation memory by ~9×.
Napkin Math: Per-GPU with ZeRO-3 (resident): sharded optimizer ≈ 70B × 12 / 8 = 105 GB — too much if fully materialized, but ZeRO-3 streams shards. Peak resident model state ≈ 20–25 GB (one layer's params gathered at a time). Activations without checkpointing (batch=1, seq=2048): ~60 GB. With checkpointing every 9 layers: ~7 GB. Total per-GPU: 25 + 7 + overhead ≈ 35–40 GB. Fits in 80 GB with room for micro-batch size 2–4. The design knob: checkpoint every $N$ layers, where $N$ balances recomputation overhead (33% at $N = \sqrt{L}$) against activation memory.
Key Equation: $\text{Activation memory} \propto \frac{L}{N} \times B \times S \times d_{\text{model}}$ where $N$ = checkpoint interval; optimal $N = \sqrt{L}$ minimizes $\text{memory} \times \text{recompute}$
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "A new hire asks you: 'FP16 and BF16 are both 16 bits. Why does everyone use BF16 for training? Aren't they the same?' Give them the 2-minute explanation."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "BF16 is just a newer, better version of FP16" or "They're basically interchangeable — BF16 is just what NVIDIA recommends now." Both miss the fundamental bit-layout trade-off.
Realistic Solution: FP16 and BF16 allocate their 16 bits differently, and that difference is existential for training stability. FP16 uses 5 exponent bits and 10 mantissa bits: high precision (~3 decimal digits) but narrow range (±65,504). BF16 uses 8 exponent bits and 7 mantissa bits: lower precision (~2 decimal digits) but the same range as FP32 (±3.4 × 10³⁸). For training, range matters more than precision because gradients can span dozens of orders of magnitude. FP16's narrow range causes small gradients to underflow to zero, requiring complex loss scaling. BF16 "just works" because its 8 exponent bits match FP32's dynamic range.
Napkin Math: FP16 min subnormal ≈ $5.96 \times 10^{-8}$, max = 65,504. BF16 min subnormal ≈ $9.18 \times 10^{-41}$, max ≈ $3.39 \times 10^{38}$. Gradients in layer 80 of a deep Transformer routinely hit $10^{-12}$ to $10^{-20}$. In FP16: underflow to zero → training collapses. In BF16: represented exactly (within precision) → training proceeds. The precision loss (10 → 7 mantissa bits) is irrelevant because SGD's stochastic noise already exceeds 2 decimal digits of precision.
Key Equation: $\text{Dynamic range} = 2^{2^{e-1}}$ where $e$ = exponent bits. FP16: $2^{16} = 65536$. BF16: $2^{128} \approx 3.4 \times 10^{38}$.
📖 Deep Dive: Volume I: Neural Computation
</details>Interviewer: "We're pre-training a 13B model. It trains perfectly in BF16 for 100k steps. When we switch to FP16 with dynamic loss scaling, it diverges around step 50k with loss spikes and eventual NaNs. The hyperparameters are identical. What is happening at step 50k that wasn't happening at step 1k?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The loss scaler isn't tuned properly — just increase the scaling factor" or "FP16 training is inherently unstable at scale." The first treats the symptom; the second is defeatist and wrong.
Realistic Solution: Early in training, gradients are large (the model is far from any minimum) and comfortably within FP16's representable range. As training converges, gradient magnitudes shrink — by step 50k, many gradients have decayed below FP16's minimum representable value of ~$5.96 \times 10^{-8}$. Dynamic loss scaling tries to compensate by multiplying the loss by a large factor before backprop, but this is a balancing act: scale too high and activations overflow to Inf; scale too low and gradients underflow to zero. Around step 50k, the gradient distribution straddles FP16's narrow range — some values need high scaling (small gradients) while others need low scaling (large activations). No single scale factor works for all parameters simultaneously, and training oscillates between overflow and underflow until it diverges. BF16 never has this problem because its 8 exponent bits give the same range as FP32.
Napkin Math: At step 1k: gradient magnitudes ≈ $10^{-3}$ to $10^{-1}$ — well within FP16's range. At step 50k: gradient magnitudes ≈ $10^{-6}$ to $10^{-12}$. FP16 floor = $5.96 \times 10^{-8}$. Everything below that floor → zero. With 70% of gradients underflowing, the effective learning rate collapses for most parameters while a few with large gradients still update — the model destabilizes. BF16 floor = $9.18 \times 10^{-41}$ — gradients of $10^{-12}$ are represented with no issue.
Key Equation: $\text{Loss scale} \times \nabla_\theta \mathcal{L} \in [\text{FP16}{\min},, \text{FP16}{\max}]$ must hold for all parameters simultaneously — an increasingly impossible constraint as gradient variance grows during training
📖 Deep Dive: Volume I: Neural Computation
</details>Interviewer: "The PM just walked over and asked: 'How much will it cost to fine-tune Llama-2-13B on our 1M-example dataset?' They need a number by end of day. Walk me through how you'd estimate this on a napkin."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "I'd need to run a benchmark first — I can't estimate without profiling." In an interview (and in real planning), you must be able to estimate from first principles. Another common mistake: forgetting to multiply by 3 for the backward pass (it's ~2× the forward pass cost).
Realistic Solution: The standard approximation for training FLOPs is $6 \times N$ FLOPs per token, where $N$ is the parameter count. The factor of 6 comes from: 2× for the forward pass (multiply-accumulate = 2 ops per parameter per token) and 4× for the backward pass (2× for gradient w.r.t. activations + 2× for gradient w.r.t. weights). Multiply by total tokens, divide by GPU throughput at realistic utilization, convert to hours, multiply by price.
Napkin Math: Params = 13B. Dataset = 1M examples × 512 avg tokens = 512M tokens. Total FLOPs = $6 \times 13 \times 10^9 \times 512 \times 10^6$ = $3.99 \times 10^{19}$ FLOPs. H100 at ~50% MFU = 989 × 0.5 = 495 TFLOPS effective. Time = $3.99 \times 10^{19} / 4.95 \times 10^{14}$ = 80,600 seconds ≈ 22.4 GPU-hours. At $2.50/GPU-hr (on-demand H100) = $56. On spot instances at $0.90/hr = $20. The PM's answer: "Under $60 on-demand, under $25 on spot."
Key Equation: $\text{GPU-hours} = \frac{6 \times N \times T}{\text{GPU}_{\text{TFLOPS}} \times \text{MFU} \times 3600 \times 10^{12}}$ where $N$ = params, $T$ = tokens, MFU = model FLOP utilization
📖 Deep Dive: Volume I: Model Training
</details>Interviewer: "The team is excited: they switched from a dense 7B model to a 47B Mixture-of-Experts model with 8 experts, 2 active per token. 'Same inference speed, 7× more knowledge!' they claim. Then they try to deploy it on a single A100 40 GB and it OOMs. What did they get wrong?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "MoE only activates 2 of 8 experts, so memory usage should be similar to a 12B dense model (6B active params × 2 for FP16)." This confuses compute with memory.
Realistic Solution: MoE saves compute but not memory. During inference, only 2 experts are active per token (~6B active parameters), so the FLOPs per token are comparable to a ~6B dense model. But all 47B parameters must reside in VRAM because the router can select any expert for any token — you don't know which experts will be needed until runtime. The memory footprint is the full 47B × 2 bytes = 94 GB in FP16, regardless of how many experts are active. This is the MoE Memory Trap: the compute profile of a 6B model with the memory profile of a 47B model.
Napkin Math: Dense 7B: FLOPs/token ≈ $2 \times 7\text{B}$ = 14 GFLOPs. Memory = 7B × 2 = 14 GB. MoE 47B (2/8 active): FLOPs/token ≈ $2 \times 6\text{B}$ = 12 GFLOPs (similar). Memory = 47B × 2 = 94 GB (6.7× more). The 47B MoE needs at least 2× A100 80 GB with tensor parallelism, or quantization to INT4 (47B × 0.5 = 23.5 GB) to fit on a single GPU. Compute is cheap; memory is the constraint.
Key Equation: $\text{VRAM}{\text{MoE}} = N{\text{total}} \times b_w$ (not $N_{\text{active}} \times b_w$)
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "We're deploying a model that supports 128k context. Users love it, but our serving costs are 10× higher than the 4k-context version, and P99 latency is through the roof. The model has the same number of parameters. Why does 32× more context cost 10× more to serve, and what are our architectural options?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Context length only affects KV-cache memory, so we just need more VRAM." Memory is part of it, but the compute cost of attention itself is the dominant factor at 128k.
Realistic Solution: Standard self-attention is $O(n^2)$ in both compute and memory with respect to sequence length. Going from 4k to 128k tokens is a 32× increase in $n$, which means a $32^2 = 1024\times$ increase in attention FLOPs and memory. The attention matrix alone at 128k is enormous. Three architectural mitigations exist, each with different trade-offs: (1) FlashAttention — eliminates the $O(n^2)$ memory by tiling to SRAM, but compute is still $O(n^2)$; it buys you memory headroom, not compute savings. (2) Sliding window / local attention — reduces to $O(n \times w)$ where $w$ is the window size, but sacrifices global context; tokens beyond the window are invisible. (3) Sparse attention with global sentinel tokens — most tokens attend locally, but a small set of global tokens attend to everything, preserving long-range dependencies at sub-quadratic cost.
Napkin Math: Attention FLOPs per layer per head at 4k tokens: $2 \times 4096^2 \times 128$ = 4.3 GFLOPs. At 128k tokens: $2 \times 131072^2 \times 128$ = 4.4 TFLOPs — a 1024× increase. Attention matrix memory at 128k in FP16: $131072^2 \times 2$ = 32 GB per head per layer. With 32 heads and 80 layers: 32 × 80 × 32 GB = 81.9 TB — obviously impossible without FlashAttention's tiling. Sliding window with $w = 4096$: attention FLOPs drop back to $2 \times 131072 \times 4096 \times 128$ = 137 GFLOPs per layer per head — a 32× reduction from full attention.
Key Equation: $\text{Attention FLOPs} = O(n^2 \cdot d)$ for full attention; $O(n \cdot w \cdot d)$ for sliding window, where $w \ll n$
📖 Deep Dive: Volume I: Neural Network Architectures
</details>Interviewer: "Facilities just told us our new rack has a 40 kW power budget. The H100 SXM has a 700W TDP. How many GPUs can we fit in this rack?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "40,000 W / 700 W = 57 GPUs." This treats the GPU as the only power consumer in the rack.
Realistic Solution: A GPU doesn't run in a vacuum. Each GPU sits in a server node that also draws power for CPUs, DRAM, NVLink switches, NVMe storage, cooling fans, and network interface cards. A typical DGX H100 node (8 GPUs) draws ~10.2 kW total: 5.6 kW for GPUs + 4.6 kW for everything else — roughly 575W overhead per GPU. Add top-of-rack networking switches (~2 kW per rack). The real calculation must account for the full system power, not just the accelerator TDP.
Napkin Math: Per-GPU system power = 700W (GPU) + ~275W (CPU/RAM/NIC/fan share) = ~975W. Rack networking overhead = ~2 kW. Usable rack power = 40 - 2 = 38 kW. GPUs per rack = $38000 / 975$ ≈ 39 GPUs (4 DGX nodes × 8 GPUs + room for 7 more, but DGX nodes come in 8-GPU units, so realistically 32 GPUs = 4 nodes). Naive estimate of 57 was off by 44%. For next-gen B200 at 1000W TDP, the gap widens further.
Key Equation: $N_{\text{GPU}} = \frac{P_{\text{rack}} - P_{\text{network}}}{\text{TDP}{\text{GPU}} + P{\text{host-per-GPU}}}$
📖 Deep Dive: Volume II: Compute Infrastructure
</details>Interviewer: "Our CFO asks: 'We're spending $86M a year on electricity for our 10,000-GPU training cluster. The facilities team says they can reduce PUE from 1.4 to 1.1 by switching to liquid cooling, but the retrofit costs $5M. Is it worth it?'"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "PUE is a facilities metric — it doesn't affect ML engineering decisions." In reality, PUE directly determines your electricity bill and can make or break the ROI of a training cluster.
Realistic Solution: PUE (Power Usage Effectiveness) is the ratio of total facility power to IT equipment power. A PUE of 1.4 means for every 1W of compute, you pay for 1.4W total — the extra 0.4W goes to cooling, lighting, and power distribution. For a 10,000-GPU cluster, that 0.4W multiplier translates to millions of dollars per year. The CFO's question is a straightforward ROI calculation: compute the annual savings from the PUE reduction and compare to the retrofit cost.
Napkin Math: 10,000 H100s × 700W = 7 MW IT load. At PUE 1.4: total = 7 × 1.4 = 9.8 MW. At PUE 1.1: total = 7 × 1.1 = 7.7 MW. Savings = 2.1 MW. Annual electricity savings = 2.1 MW × 8,760 hrs × $0.10/kWh = $1.84M/year. $5M retrofit payback period = $5M / $1.84M ≈ 2.7 years. For a cluster with a 5-year lifespan, total savings = $1.84M × 5 - $5M = $4.2M net. Tell the CFO: "Yes, the retrofit pays for itself in under 3 years."
Key Equation: $\text{Annual cost} = P_{\text{IT}} \times \text{PUE} \times 8760 \times C_{\text{kWh}}$ and $\Delta\text{Cost} = P_{\text{IT}} \times (\text{PUE}{\text{old}} - \text{PUE}{\text{new}}) \times 8760 \times C_{\text{kWh}}$
📖 Deep Dive: Volume II: Compute Infrastructure
</details>Interviewer: "We have two identical 1,000-GPU H100 clusters — same hardware, same software, same model. Cluster A in Phoenix, Arizona consistently trains 30% slower than Cluster B in The Dalles, Oregon. The ops team has checked everything: drivers, firmware, network config — all identical. What's going on?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "There must be a software misconfiguration or a bad batch of GPUs" or "Network latency between racks is higher in Phoenix." Both ignore the physical environment the hardware sits in.
Realistic Solution: Thermal throttling. Phoenix ambient temperature in summer reaches 45°C (113°F). The Dalles averages 20°C (68°F). Data center cooling systems have finite capacity — they can only reject a certain number of watts of heat to the outside air. When ambient temperature rises, the temperature delta between the coolant and the outside air shrinks, reducing cooling effectiveness. When GPU junction temperatures exceed ~83°C, the hardware automatically reduces clock speeds and power draw (from 700W to ~500W) to prevent damage. This is thermal throttling, and it directly reduces training throughput. The 30% slowdown maps almost exactly to the power reduction: $500/700 = 71%$ of peak performance.
Napkin Math: H100 TDP = 700W, throttle point = ~83°C junction. Phoenix data center: 45°C ambient → cooling struggles → GPU junction hits 83°C → throttles to 500W → 71% of peak throughput. Oregon: 20°C ambient → GPU junction stays at ~65°C → full 700W → 100% throughput. The 29% gap matches the reported 30% slowdown. Fix options: (1) liquid cooling (removes ambient dependency), (2) schedule heavy training jobs at night (Phoenix drops to 25°C), (3) over-provision cooling capacity (expensive). This is why hyperscalers build in Oregon, Iowa, and the Nordics — not Phoenix.
Key Equation: $P_{\text{cooling}} = \dot{m} \times c_p \times (T_{\text{coolant}} - T_{\text{ambient}})$ — as $T_{\text{ambient}} \uparrow$, cooling capacity $\downarrow$ at fixed infrastructure
📖 Deep Dive: Volume II: Compute Infrastructure
</details>Interviewer: "You build a new cluster with 1,000 H100 GPUs. The facility has a strict 1 Megawatt (1MW) power limit. You calculate that each GPU uses 700W, so 1000 * 700W = 700kW. You assume you have 300kW of headroom for CPUs and networking. Yet, the moment you launch a massive distributed AllReduce, the main datacenter breaker trips and the entire cluster goes dark. What macro-physics did you miss?"
Common Mistake: "Treating Thermal Design Power (TDP) as a flat, continuous draw, and forgetting the infrastructure required to move the heat away from the chips."
Realistic Solution: You ignored Power Usage Effectiveness (PUE) and transient power spikes. First, TDP is an average. During intense synchronized operations like AllReduce, GPUs can experience millisecond-level power excursions (spikes) that pull 1.5x to 2x their rated TDP. If 1,000 GPUs spike simultaneously, your 700kW draw instantly spikes to 1.2MW. Second, PUE accounts for cooling. A typical PUE is 1.2, meaning for every 1 Watt of compute, you need 0.2 Watts to run the CRAC units, chillers, and pumps. 700kW of compute requires 140kW of cooling. You mathematically exceeded the facility's physical limit before the job even started.
Napkin Math:
Compute Power = 1000 GPUs * 700W = 700kW.Facility Overhead (PUE 1.2) = 700kW * 1.2 = 840kW. Add CPUs, switches, and optics (~150kW) ->990kW. You are right at the 1MW edge. When the distributed sync happens, a 20% transient power spike hits990kW * 1.2 = ~1.18 MW. The main breaker physically trips to prevent the wires from melting.
📖 Deep Dive: Volume I: ML Operations
</details>Interviewer: "You are deploying a CPU-based embedding retrieval system on an 8-socket, 128-core enterprise server. When you run 1 instance of the app, throughput is 10,000 QPS. To scale up, you spawn 8 independent instances of the app using Docker. Instead of 80,000 QPS, total throughput is only 30,000 QPS. What physical motherboard bottleneck are you hitting?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Treating a massive multi-socket server as one giant, flat CPU, ignoring Non-Uniform Memory Access (NUMA) domains."
Realistic Solution: You are suffering from severe NUMA cross-talk. In massive servers, CPUs are physically separated into distinct sockets, and each socket has its own dedicated banks of RAM attached directly to it. If Docker process 1 is scheduled on CPU Socket A, but the OS allocator happens to put its embedding tables in the RAM attached to CPU Socket D, every single memory read must travel across the motherboard interconnect (like Intel UPI or AMD UPI). This physically chokes the interconnect bandwidth and adds massive latency.
Napkin Math: Local memory access (within the same NUMA node) might take
~80 nanosecondswith~100 GB/sbandwidth. Remote memory access (crossing to another socket's RAM) might take~140 nanosecondswith bandwidth choked to~20 GB/sdue to link saturation. By not pinning (affinitizing) your processes and memory to specific NUMA nodes (e.g., usingnumactl), your 8 instances are blindly thrashing data back and forth across the motherboard.
📖 Deep Dive: Volume I: Hardware Acceleration
</details>Interviewer: "You are building an 8-GPU server for training. You buy 8x H100 GPUs and a dual-socket CPU motherboard. You plug 4 GPUs into the PCIe slots wired to CPU 0, and 4 GPUs into the slots wired to CPU 1. When you run NCCL AllReduce to sync gradients, the training is 3x slower than the same 8 GPUs running in an official Nvidia DGX system. nvidia-smi shows the GPUs are mostly idle. Where is the bottleneck?"
Common Mistake: "Assuming that as long as all GPUs are plugged into the same motherboard, they can talk to each other at full PCIe speeds."
Realistic Solution: You forced GPU-to-GPU traffic over the slow CPU interconnect. In a dual-socket system, the PCIe lanes are split between the two CPUs. GPU 0 (on CPU 0) can talk to GPU 3 (on CPU 0) directly via a PCIe switch at high speed. However, for GPU 0 to send a gradient to GPU 4 (on CPU 1), the data must physically travel from GPU 0, into CPU 0, across the QPI/UPI (Ultra Path Interconnect) bridge to CPU 1, and then down to GPU 4. The UPI bridge is drastically slower than PCIe/NVLink and becomes an extreme choke point for ring or tree AllReduce topologies.
Napkin Math: A standard PCIe Gen 5 x16 slot provides
~64 GB/sof bidirectional bandwidth. An Nvidia NVLink switch (like in a DGX) provides900 GB/sof GPU-to-GPU bandwidth. The Intel UPI link connecting two CPUs might only provide~20-40 GB/sof practical throughput, and it is shared with all other system traffic. Your communication bottleneck is the physical wire between the two CPU sockets.
📖 Deep Dive: Volume I: Hardware Acceleration
</details>Interviewer: "Your team is serving Llama-2-70B. On NVIDIA H100 80 GB, you need 2-way tensor parallelism — the 140 GB of FP16 weights simply don't fit on one GPU. AMD just offered you MI300X nodes with 192 GB HBM3 per GPU. The sales rep says 'you can serve 70B on a single GPU now.' Is this true, and what's the real systems impact of going from 2 GPUs to 1?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "More memory just means we can serve larger models — the latency and throughput stay the same." This ignores the cascading systems effects of eliminating tensor parallelism.
Realistic Solution: On H100, Llama-2-70B in FP16 requires 140 GB for weights alone. With KV-cache for even a modest batch, you exceed 80 GB and must shard across 2 GPUs using tensor parallelism (TP=2). TP introduces two costs: (1) an AllReduce after every attention and FFN block — 160 AllReduces per forward pass for an 80-layer model, each paying NVLink latency (~5 μs) plus transfer time; (2) you halve the effective memory bandwidth because each GPU only holds half the weights but must synchronize at every layer. On MI300X with 192 GB HBM3, the 140 GB of weights fit on a single GPU with 52 GB remaining for KV-cache and activations. Eliminating TP removes all inter-GPU communication, reduces tail latency variance (no synchronization jitter), and the MI300X's 5.3 TB/s HBM3 bandwidth (vs H100's 3.35 TB/s) directly accelerates the memory-bound decode phase.
Napkin Math: H100 TP=2: Weights = 140 GB sharded to 70 GB/GPU. KV-cache per request (4096 tokens, 80 layers, 64 heads, d=128, FP16) = $2 \times 80 \times 64 \times 128 \times 4096 \times 2$ ≈ 10.7 GB. Max batch on 80 GB: $(80 - 70) / 10.7 \approx$ 0.9 — barely 1 request per GPU before OOM. Communication overhead: 160 AllReduces × ~10 μs each = 1.6 ms added latency per token. MI300X TP=1: Weights = 140 GB on 1 GPU. Free memory = 192 - 140 = 52 GB. Max batch = $52 / 10.7 \approx$ 4 concurrent requests with no communication overhead. Decode bandwidth: MI300X at 5.3 TB/s reads 140 GB of weights in 26.4 ms; H100 at 3.35 TB/s reads 70 GB (sharded) in 20.9 ms + sync overhead ≈ 22.5 ms. Single-GPU MI300X is only 17% slower per token but serves 4× the batch — throughput per dollar shifts dramatically.
📖 Deep Dive: Volume I: Hardware Acceleration
</details>Interviewer: "We're planning a datacenter refresh: replacing our A100 cluster with NVIDIA B200 GPUs. The B200 delivers 4.5× the FP8 TFLOPS of the A100, but its TDP is 1000W versus the A100's 400W. Our facilities team says the existing 40 kW racks and cooling can handle it. Should we trust them?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "We just need fewer GPUs to get the same throughput, so total rack power stays the same." This assumes you'll voluntarily leave GPU slots empty — in practice, teams fill every slot to maximize throughput.
Realistic Solution: The power wall is real and multi-dimensional. A100 nodes (8 GPUs) draw ~5 kW for GPUs + ~2.5 kW host overhead = ~7.5 kW per node. A 40 kW rack fits 4 nodes (32 GPUs) at ~30 kW IT load + networking. B200 nodes (8 GPUs) draw ~8 kW for GPUs + ~3 kW host overhead = ~11 kW per node. At 40 kW, you can only fit 3 nodes (24 GPUs) — you've lost 25% of your GPU count per rack. Worse, the cooling infrastructure designed for 30 kW of heat rejection per rack now faces 33+ kW. Air cooling at this density hits its physical limit (~35-40 kW/rack); you likely need direct-to-chip liquid cooling, which requires plumbing retrofits costing $50-100K per rack. The "4.5× compute" upgrade actually requires rethinking the entire physical plant.
Napkin Math: A100 rack (40 kW budget): 4 nodes × 8 GPUs = 32 GPUs. IT power = 32 × 400W + host overhead = ~15.3 kW. With PUE 1.3: 15.3 × 1.3 = 19.9 kW — comfortable. B200 rack (40 kW budget): Per-GPU system power = 1000W GPU + ~375W host share = 1375W. GPUs per rack = $(40000 / 1.3) / 1375 \approx$ 22 GPUs (2 full nodes + partial). Heat density = 22 × 1375 = 30.3 kW IT → 30.3 × 1.3 = 39.4 kW total — at the absolute limit. Air cooling capacity for a standard 42U rack tops out at ~35 kW; you need liquid cooling. Aggregate impact: A 1000-GPU A100 cluster = 32 racks. Equivalent B200 compute (1000/4.5 ≈ 222 GPUs) = 10 racks — but if you fill to 1000 B200s for max throughput, you need 46 racks with liquid cooling infrastructure.
📖 Deep Dive: Volume II: Compute Infrastructure
</details>Interviewer: "Your company is choosing between a Google Cloud TPU v5e pod (256 chips) and an equivalent NVIDIA H100 cluster for serving a 7B parameter model at 50,000 requests per second. The TPU v5e costs $1.20/chip-hour and the H100 costs $3.50/GPU-hour. The PM says 'TPU is obviously cheaper.' Walk me through why this isn't a simple price comparison."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just compare $/chip-hour — TPU is 3× cheaper per chip, so it wins." This ignores that the chips have fundamentally different architectures, memory capacities, and software ecosystems, making per-chip comparison meaningless.
Realistic Solution: The correct metric is cost per token (or cost per request at a given latency SLO), not cost per chip. TPU v5e has 16 GB HBM per chip — a 7B FP16 model (14 GB) barely fits on one chip with almost no room for KV-cache. You need multi-chip sharding even for a 7B model at reasonable batch sizes. H100 has 80 GB HBM — the 7B model fits trivially with 66 GB free for batching. The TPU v5e compensates with its ICI (Inter-Chip Interconnect) at 1.6 Tbps per chip, enabling efficient sharding across the pod. But the H100's advantage is software maturity: CUDA kernels (FlashAttention, PagedAttention) are battle-tested, while TPU XLA compilation can leave 20-30% performance on the table for serving workloads with dynamic shapes. The real trade-off: TPU v5e wins on large-batch throughput-optimized serving (where ICI bandwidth amortizes sharding cost); H100 wins on latency-sensitive serving with diverse request patterns.
Napkin Math: H100 serving 7B model: Weights = 14 GB. Free for KV-cache = 66 GB. KV-cache per request (2048 tokens) ≈ 0.5 GB. Max batch ≈ 132 concurrent requests. Decode throughput (bandwidth-bound): 3.35 TB/s / 14 GB = 239 tokens/s/request at batch=1; at batch=132, ~1.8 tokens/s/request but 239 tokens/s aggregate. Cost: 1 GPU at $3.50/hr serving ~860K tokens/hr = $4.07 per million tokens. TPU v5e serving 7B model: Need TP=2 (16 GB/chip too tight). 2 chips serve 1 model replica. Effective bandwidth = 819 GB/s per chip. Decode: 819 GB/s / 7 GB (sharded) = 117 tokens/s per replica. 128 replicas on 256-chip pod = ~15K tokens/s. Cost: 256 × $1.20 = $307/hr for ~54M tokens/hr = $5.69 per million tokens. TPU v5e is cheaper per chip but more expensive per token for this model size. The crossover happens at larger models (70B+) where ICI-connected pods amortize the sharding overhead.
📖 Deep Dive: Volume I: Hardware Acceleration
</details>Interviewer: "Intel is pitching us Gaudi 3 accelerators for our training cluster. Instead of hand-written CUDA kernels, Gaudi uses a graph compiler that automatically fuses and schedules operations. The sales team claims 'you get kernel-level performance without writing kernels.' Your CUDA team is skeptical. What are the real systems trade-offs?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "A compiler can never match hand-tuned CUDA kernels, so Gaudi will always be slower." This underestimates modern graph compilers and ignores the total cost of ownership, including engineering time.
Realistic Solution: The trade-off is between peak performance and development velocity. CUDA gives you direct control over shared memory tiling, warp scheduling, and register allocation — expert kernel engineers can extract 80-90% of theoretical peak. Gaudi's graph compiler (SynapseAI) operates at a higher abstraction: it takes a PyTorch graph, performs operator fusion, memory planning, and instruction scheduling automatically. For standard operations (GEMM, attention, LayerNorm), the compiler achieves 70-85% of what a hand-tuned kernel would deliver. The gap is real but narrow for common patterns. Where the compiler wins: (1) novel architectures — when you change your model, the compiler re-optimizes automatically, while CUDA requires weeks of kernel re-engineering; (2) fusion opportunities — the compiler can fuse chains of operations that no pre-written kernel library covers; (3) engineering cost — a Gaudi deployment needs 2-3 ML engineers, while a CUDA deployment at the same scale needs 2-3 ML engineers plus 1-2 kernel specialists at $400K+/year. Where the compiler loses: (1) tail operations with irregular memory access patterns; (2) workloads requiring custom memory management (like PagedAttention); (3) debugging — when the compiler generates slow code, you have limited visibility into why.
Napkin Math: Gaudi 3 specs: 1835 TFLOPS BF16, 128 GB HBM2e at 3.7 TB/s. H100 SXM: 989 TFLOPS BF16 (1979 with sparsity), 80 GB HBM3 at 3.35 TB/s. Raw BF16 TFLOPS favors Gaudi 3 by ~1.85×. But MFU (Model FLOP Utilization) matters: H100 with optimized CUDA achieves 55-65% MFU on LLM training; Gaudi 3 with compiler typically achieves 45-55% MFU. Effective throughput: H100 = 989 × 0.60 = 593 TFLOPS; Gaudi 3 = 1835 × 0.50 = 918 TFLOPS. Gaudi 3 still leads on effective throughput despite lower MFU, because the raw silicon advantage is large enough. The real question is $/TFLOP-effective including engineering costs over a 3-year deployment.
📖 Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Our 70B model serves chat completions with a P50 time-to-first-token of 200 ms and a P50 inter-token latency of 45 ms on H100. Product wants 2× faster decoding without changing the model. Someone suggests speculative decoding with a 1B draft model. Walk me through the systems math — when does this help, and when does it backfire?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The draft model is 70× smaller, so it generates tokens 70× faster, and we just verify them in parallel — easy 5-10× speedup." This ignores the acceptance rate, the memory overhead of running two models, and the verification cost.
Realistic Solution: Speculative decoding works in three steps: (1) the draft model generates $K$ candidate tokens autoregressively (cheap, ~1 ms/token for a 1B model); (2) the target 70B model verifies all $K$ tokens in a single forward pass (parallel verification — same cost as generating 1 token, since decode is memory-bandwidth-bound and the extra compute for $K$ tokens is negligible); (3) the target accepts the first $n \leq K$ tokens where the draft model's distribution matches, and resamples the $(n+1)$-th token. The speedup depends critically on the acceptance rate $\alpha$ — the probability the draft model's token matches the target's. If $\alpha = 0.8$ and $K = 5$, the expected accepted tokens per verification step is $\sum_{i=0}^{K} \alpha^i \approx 1/(1-\alpha) = 5$ tokens per target forward pass (in the limit). But you also pay for the draft model's memory and compute. If the draft model's memory displaces KV-cache space, your maximum batch size drops, reducing throughput even as per-request latency improves.
Napkin Math: Without speculation: 70B decode = 45 ms/token (bandwidth-bound: 140 GB weights / 3.35 TB/s ≈ 42 ms + overhead). With speculation (K=5, α=0.8): Draft generates 5 tokens: 5 × 1 ms = 5 ms. Target verifies: 45 ms (one forward pass). Expected accepted tokens: $1/(1-0.8) = 5$ tokens. Effective per-token latency: $(5 + 45) / 5 = $ 10 ms/token — a 4.5× speedup. Memory cost: 1B draft model = 2 GB + its KV-cache ≈ 2.5 GB. On 80 GB H100 with 70B model (140 GB sharded TP=2 → 70 GB/GPU), free memory drops from 10 GB to 7.5 GB — max batch drops from ~9 to ~7 requests. When it backfires: If $\alpha$ drops to 0.4 (e.g., code generation where the draft model is weak), expected accepted = $1/0.6 \approx 1.67$ tokens per step. Effective latency: $(5 + 45) / 1.67 = $ 30 ms/token — only 1.5× speedup, and you've lost 25% of your batch capacity for it.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "We're training a 175B model on 10,000 H100 GPUs. At step 50,000, a node fails and the job crashes. We checkpoint every 1,000 steps. The PM asks: 'We only lost 1,000 steps of work, right? So we restart and lose maybe 30 minutes?' Explain why the PM's estimate is dangerously optimistic."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Checkpoint overhead is negligible — just reload the last checkpoint and resume." This ignores the time to write checkpoints, the time to restart 10,000 GPUs, and the cascading costs of failure at scale.
Realistic Solution: The PM's estimate misses four costs: (1) Checkpoint write time — a 175B model in mixed-precision training has ~2.8 TB of state (weights + optimizer + gradients). Writing this to distributed storage (even parallel across nodes) takes significant time. (2) Lost compute — the 1,000 steps between checkpoints represent real GPU-hours that are irrecoverable. (3) Restart overhead — re-initializing 10,000 GPUs, re-establishing NCCL communication rings, loading the checkpoint from storage, and verifying consistency takes 15-45 minutes. (4) Checkpoint loading — reading 2.8 TB from distributed storage back into 10,000 GPUs is not instant. The total recovery time is far longer than "30 minutes," and the dollar cost of lost compute is substantial.
Napkin Math: Checkpoint size: 175B params × 16 bytes (FP16 weights + FP32 master + FP32 Adam m,v) = 2.8 TB. Write time: Parallel write across 1,250 nodes to a distributed filesystem at ~10 GB/s aggregate = 2800 / 10 = ~280 seconds (~4.7 min) per checkpoint. At every 1,000 steps, this is a 4.7-minute pause every ~30 minutes of training — ~14% overhead just for checkpointing. Lost compute: 1,000 steps × 10,000 GPUs × ~45 ms/step = 450,000 GPU-seconds = 125 GPU-hours. At $3.50/GPU-hr = $437 of lost compute. Restart cost: Job scheduler queue wait: ~5 min. NCCL initialization for 10,000 GPUs: ~10 min. Checkpoint load (2.8 TB from storage): ~5 min. Warmup/verification: ~5 min. Total restart: ~25 minutes of 10,000 idle GPUs = 4,167 GPU-hours = $14,583. The restart cost dwarfs the lost compute. Over a 90-day training run with MTBF of ~8 hours per failure at this scale, expect ~270 failures. Total failure cost: 270 × ($437 + $14,583) = $4.1M — a line item that must be budgeted.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We're training a 13B model across 128 GPUs connected by 400 Gbps InfiniBand. The network is the bottleneck — AllReduce takes 40% of each step. An engineer proposes gradient compression with a 100× compression ratio. They claim this will reduce communication time by 100×, making the network overhead negligible. Why won't they get anywhere near 100× improvement?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "100× compression means 100× less data to send, so communication time drops by 100×." This treats the network as the only cost and ignores compression/decompression overhead, latency-bound operations, and convergence effects.
Realistic Solution: Three factors conspire to destroy the theoretical 100× speedup: (1) Compression and decompression compute cost — algorithms like TopK or random sparsification require sorting or sampling all gradients on the GPU before sending, and reconstruction on the receiving end. This adds GPU compute that partially offsets the bandwidth savings. (2) Latency dominance — AllReduce has two components: bandwidth term (data volume / link bandwidth) and latency term (number of synchronization steps × per-step latency). At 100× compression, the bandwidth term shrinks to near zero, but the latency term is unchanged — you still need $2(p-1)$ sequential steps in ring AllReduce, each paying ~5 μs network latency + ~10 μs kernel launch overhead. At 128 GPUs, that's $254 \times 15 \mu s = 3.8$ ms of irreducible latency. (3) Convergence degradation — aggressive compression introduces gradient noise. To converge to the same loss, you typically need 1.3-2× more training steps, clawing back much of the wall-clock savings.
Napkin Math: Uncompressed AllReduce: 13B params × 4 bytes (FP32 gradients) = 52 GB. Ring AllReduce bandwidth time: $2 \times 52\text{ GB} / (400\text{ Gbps} / 8) = 2 \times 52 / 50 = $ 2.08 seconds. Latency: $2 \times 127 \times 15 \mu s = $ 3.8 ms. Total ≈ 2.08 s (bandwidth-dominated). 100× compressed AllReduce: Bandwidth time: $2.08 / 100 = $ 20.8 ms. Latency: still 3.8 ms. Compression overhead (TopK sort on 13B elements): ~15 ms per GPU. Decompression: ~8 ms. Total ≈ $20.8 + 3.8 + 15 + 8 = $ 47.6 ms. Actual speedup: $2080 / 47.6 = $ 43.7× — not 100×. Factor in 1.5× more steps to converge: effective speedup = $43.7 / 1.5 = $ 29×. Significant, but a far cry from the promised 100×.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We serve a 70B model on a fleet of 64 H100 GPUs. During peak traffic, our P99 latency spikes to 5× the P50 because long-prompt requests (8k+ tokens) block short-prompt requests in the same batch. An architect proposes splitting the fleet into separate 'prefill GPUs' and 'decode GPUs.' This sounds like it wastes resources. Why would dedicating GPUs to different phases actually improve both throughput and latency?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Splitting the fleet means each pool has fewer GPUs, so both pools will be slower." This applies static capacity thinking to a problem that's fundamentally about workload heterogeneity.
Realistic Solution: Prefill and decode have opposite computational profiles, and mixing them on the same GPU creates destructive interference. Prefill (processing the input prompt) is compute-bound: it processes all prompt tokens in parallel through matrix multiplications. Optimal batch size is small (1-4 requests) because each request consumes massive FLOPs. Decode (generating output tokens) is memory-bandwidth-bound: it reads the entire model weights to produce one token per request. Optimal batch size is large (64-256 requests) to amortize the weight-reading cost. When both phases share a GPU, you can't optimize for either: large decode batches starve prefill of compute, while prefill's heavy compute blocks decode tokens from being emitted, causing latency spikes. Disaggregation lets each pool run at its optimal operating point. Prefill GPUs run small batches at high compute utilization; decode GPUs run large batches at high bandwidth utilization. A lightweight scheduler routes requests: new requests go to prefill GPUs, and once the KV-cache is computed, it's transferred to a decode GPU (KV-cache transfer is a one-time cost per request).
Napkin Math: 70B model, H100 (989 TFLOPS, 3.35 TB/s). Prefill (2048-token prompt): FLOPs = $2 \times 70\text{B} \times 2048 = 287$ TFLOPS. Time at 60% MFU: $287 / (989 \times 0.6) = $ 0.48 s. Arithmetic intensity = $287 \times 10^{12} / (140 \times 10^9) = 2050$ Ops/Byte — deeply compute-bound. Optimal batch: 1-2 requests. Decode (1 token): FLOPs = $2 \times 70\text{B} = 140$ GFLOPs. Weight read = 140 GB. Time (bandwidth-bound): $140 / 3350 = $ 41.8 ms. Arithmetic intensity = $140 \times 10^9 / (140 \times 10^9) = 1$ Ops/Byte — deeply memory-bound. Optimal batch: at batch=128, you do 128× more compute with the same weight read, pushing utilization from 0.04% to ~5%. Mixed fleet (64 GPUs): Prefill requests with 8k tokens take ~1.9 s, blocking decode for all co-located requests → P99 spikes. Disaggregated (20 prefill + 44 decode): Prefill GPUs: 20 GPUs handle ~42 prefill requests/s. Decode GPUs: 44 GPUs at batch=128 each handle ~3,072 tokens/s decode. KV-cache transfer (2048 tokens × 10.7 GB): ~3.2 ms over NVLink — negligible. P99 decode latency drops from ~200 ms to ~50 ms because decode GPUs never stall on prefill compute.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "We're 60 days into a 90-day training run on 10,000 H100 GPUs. The loss curve looks normal, but when we evaluate on our held-out benchmark, accuracy is 8 points below the expected scaling law prediction. No crashes, no NaNs, no obvious errors in the logs. What could cause a model to silently underperform, and how would you detect it?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The scaling law prediction must be wrong, or the benchmark is noisy." This dismisses the most dangerous failure mode in large-scale training: silent data corruption (SDC).
Realistic Solution: At 10,000-GPU scale, silent hardware errors are not rare events — they're statistical certainties. HBM bit flips, intermittent PCIe errors, and faulty Tensor Cores can corrupt individual gradient or activation values without triggering ECC errors or NaN checks. A single corrupted gradient in one AllReduce poisons the update for all GPUs. The training loss may still decrease (SGD is robust to some noise), but the model learns subtly wrong representations. Detection requires active monitoring beyond loss curves: (1) periodic evaluation on a fixed validation set (not just training loss); (2) gradient norm tracking per GPU — a GPU with faulty memory will show anomalous gradient statistics; (3) checksum-based AllReduce verification — compare AllReduce results across redundant computation paths; (4) "canary" computations — run a fixed input through the model every N steps and compare output to a known-good reference.
Napkin Math: Failure rates: Google's 2023 study reported ~1-2 silent data corruption events per 1,000 GPU-days for A100-class hardware. At 10,000 GPUs over 90 days: expected SDC events = $10000 \times 90 / 1000 \times 1.5 = $ 1,350 silent corruption events during the training run. Even if 99% are benign (corrupted values that get averaged away), 1% causing meaningful gradient corruption = ~14 events that shift the model. Each corrupted AllReduce affects all 10,000 GPUs simultaneously. Detection cost: Running a 1,000-example validation eval every 100 steps: 1,000 × $2 \times 70\text{B}$ = 140 TFLOPs per eval. At 989 TFLOPS per GPU, this takes ~0.14 seconds — negligible. Gradient norm monitoring: one
torch.norm()per GPU per step = microseconds. The monitoring overhead is <0.1% of training time; the cost of not monitoring is potentially discarding a $50M+ training run.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We scale our data-parallel training from 32 GPUs to 512 GPUs. On 32 GPUs, AllReduce takes 15% of each training step. On 512 GPUs, it takes 60%. The network hardware is the same 400 Gbps InfiniBand everywhere. Why does ring AllReduce degrade at scale, and what replaces it?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Ring AllReduce is bandwidth-optimal, so it should scale perfectly — the problem must be network congestion." Ring AllReduce is indeed bandwidth-optimal in theory, but this ignores the latency term that dominates at scale.
Realistic Solution: Ring AllReduce completes in $2(p-1)$ sequential communication steps, where $p$ is the number of GPUs. Each step pays a fixed latency cost (network hop + kernel launch + synchronization). The bandwidth term is constant regardless of $p$ (each GPU sends and receives $\text{data} \times (p-1)/p \approx \text{data}$ total), but the latency term grows linearly with $p$. At small $p$, the bandwidth term dominates and scaling looks perfect. At large $p$, the latency term dominates and each additional GPU adds pure overhead. The fix is hierarchical AllReduce: first reduce within each node (8 GPUs over NVLink at 900 GB/s — microseconds), then reduce across nodes (using a tree or recursive-halving algorithm that has $O(\log p)$ latency steps instead of $O(p)$). NCCL automatically switches to tree AllReduce at scale, but the topology must support it.
Napkin Math: Model: 7B params, FP32 gradients = 28 GB. Ring AllReduce at 32 GPUs: Bandwidth time: $2 \times 28\text{ GB} \times (31/32) / 50\text{ GB/s} = $ 1.09 s. Latency: $2 \times 31 \times 15 \mu s = $ 0.93 ms. Total ≈ 1.09 s (bandwidth-dominated). Ring AllReduce at 512 GPUs: Bandwidth time: $2 \times 28 \times (511/512) / 50 = $ 1.12 s (barely changed). Latency: $2 \times 511 \times 15 \mu s = $ 15.3 ms. Total ≈ 1.13 s. The latency overhead grew 16×, but the total only grew 4% — so where's the 60% overhead? The real killer is synchronization jitter: with 512 GPUs, the slowest GPU in each ring step determines the pace. Even 1% straggler probability per GPU means $P(\text{no straggler in 512}) = 0.99^{512} = 0.6%$ — virtually every step has a straggler. Each straggler adds ~5-20 ms. Over 1,022 sequential steps: expected straggler delay ≈ 5-10 seconds per AllReduce. Tree AllReduce: $O(\log_2 512) = 9$ steps instead of 1,022. Straggler impact drops proportionally.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We're pre-training a 30B model. It trains fine in BF16 for 200k steps. Management asks us to switch to FP8 training to get 2× throughput on H100's FP8 Tensor Cores. After switching, we see loss spikes every ~5,000 steps that gradually get worse until training diverges at step 80k. What's happening, and how do we fix it without giving up FP8?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "FP8 is just too low precision for training — we need to go back to BF16." This gives up the 2× throughput gain without understanding which specific operations are precision-sensitive.
Realistic Solution: FP8 has two formats: E4M3 (4 exponent, 3 mantissa bits — range of ±448, ~1.5 decimal digits precision) and E5M2 (5 exponent, 2 mantissa bits — range of ±57344, ~1 decimal digit). The key insight is that not all operations tolerate the same precision. Matrix multiplications (GEMMs) in the forward pass are robust to FP8 E4M3 because the accumulation happens in FP32 inside the Tensor Core — the low precision only affects inputs. But certain operations are precision-critical: (1) Softmax — involves exponentiation where small input differences create large output differences; FP8's coarse mantissa causes attention weights to "snap" to nearby values, creating systematic bias. (2) LayerNorm — the variance computation in FP8 loses small deviations, causing normalization to over-correct. (3) Residual connections — adding a small update to a large residual in FP8 causes the update to round to zero (catastrophic cancellation). The fix is mixed FP8/BF16: run GEMMs in FP8 (where 90%+ of FLOPs live) but keep softmax, LayerNorm, residual additions, and the loss computation in BF16. This captures ~80% of the FP8 throughput gain while maintaining BF16 stability.
Napkin Math: FP8 E4M3 precision: 3 mantissa bits → values are quantized to 1 of 8 levels between consecutive powers of 2. For attention logits around 1.0, the representable values are {1.0, 1.125, 1.25, 1.375, 1.5, 1.625, 1.75, 1.875}. Two logits of 1.31 and 1.37 both round to 1.375 — the softmax treats them identically, losing the model's learned distinction. Throughput breakdown: In a Transformer forward pass, GEMMs account for ~85% of FLOPs (QKV projections, attention matmul, FFN). Running GEMMs in FP8 at 1979 TFLOPS vs BF16 at 989 TFLOPS: GEMM speedup = 2×. Non-GEMM operations (15% of FLOPs) stay at BF16 speed. Amdahl's Law: overall speedup = $1 / (0.15 + 0.85/2) = $ 1.74× — still a major win over pure BF16.
📖 Deep Dive: Volume I: Neural Computation
</details>Interviewer: "We serve a 70B model with 128k context. A single request's KV-cache in FP16 is 335 GB — larger than the model itself. We can only serve 1 concurrent request per 4-GPU shard. An engineer proposes quantizing the KV-cache to INT4, claiming '4× memory savings, 4× more concurrent requests.' What's the real trade-off?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "KV-cache quantization is lossless because attention weights are robust to noise" or conversely "Any quantization of the KV-cache will destroy model quality." Both are wrong — the impact is highly non-uniform across layers and attention heads.
Realistic Solution: KV-cache quantization from FP16 to INT4 does save ~4× memory, but the accuracy impact depends on where in the sequence and which layers you compress. Recent tokens' KV entries are accessed with high attention weight — quantization errors here directly corrupt the output. Distant tokens' entries are accessed with low attention weight — quantization errors are attenuated. Similarly, lower layers (which capture syntactic patterns) are more robust to quantization than upper layers (which capture semantic reasoning). The optimal strategy is mixed-precision KV-cache: keep the most recent $W$ tokens and the first few "sink" tokens in FP16, quantize the middle tokens to INT4, and optionally evict tokens with consistently low attention scores. This achieves 2-3× effective compression with <1% accuracy degradation, versus 4× compression with 3-8% degradation from naive uniform INT4.
Napkin Math: 70B model, 128k context, FP16 KV-cache: Per request: $2 \times 80 \times 64 \times 128 \times 128000 \times 2 = $ 335 GB. On 4× H100 (320 GB total), after 140 GB weights: 180 GB free → 0.5 requests — can't even serve one. Naive INT4 KV-cache: $335 / 4 = $ 83.8 GB. Free memory: 180 - 83.8 = 96.2 GB → can serve 2 concurrent 128k requests. But accuracy drops ~5% on long-context benchmarks (RULER, Needle-in-Haystack). Mixed-precision strategy: Keep last 4k tokens + first 256 "sink" tokens in FP16 = $4256/128000 \times 335 = $ 11.1 GB in FP16. Quantize remaining 123,744 tokens to INT4 = $(123744/128000) \times 335 / 4 = $ 81.0 GB. Total: 92.1 GB per request. Free: 180 - 92.1 = 87.9 GB → 1.9 requests (round to 1 full + prefill overlap). Accuracy drop: <1% because high-attention tokens retain full precision. Effective compression: $335 / 92.1 = $ 3.6× with minimal quality loss.
📖 Deep Dive: Volume I: Model Compression
</details>Interviewer: "We run a recommendation model with a 1 TB embedding table (4 billion entries × 128 dimensions × FP16). The table is sharded across 64 GPUs, each holding ~16 GB. Training throughput is terrible — GPUs are 80% idle. The network isn't saturated. What's going on?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The embedding table is too large — we need more GPUs to reduce per-GPU memory." Adding GPUs to a poorly sharded embedding table makes the problem worse, not better.
Realistic Solution: Embedding lookups follow a Zipf distribution: a tiny fraction of entries (popular items) account for the vast majority of accesses. With naive hash-based sharding (entry $i$ goes to GPU $i \mod 64$), the popular entries are spread uniformly — every batch requires AllToAll communication to fetch embeddings from remote GPUs. Worse, the hot entries create load imbalance: the few GPUs holding the most popular entries become bottlenecks while others sit idle. The 80% idle time is GPUs waiting for AllToAll to complete. The fix is popularity-aware sharding: (1) identify the top-1% hottest entries (which serve ~50% of lookups by Zipf's law); (2) replicate these entries on every GPU — they fit easily (1% of 1 TB = 10 GB, replicated on each GPU); (3) shard the remaining 99% of cold entries across GPUs. Now 50% of lookups are local (no communication), and the remaining 50% are evenly distributed across GPUs (no hot spots).
Napkin Math: Zipf distribution: Top 1% of entries (40M entries × 256 bytes = 10.2 GB) serve ~50% of lookups. Remaining 99% (3.96B entries = 1,014 GB) serve the other 50%. Naive sharding (64 GPUs): Every batch of 65,536 lookups: ~50% hit hot entries spread across all 64 GPUs → AllToAll moves ~32,768 embeddings × 256 bytes = 8.4 MB per batch. At 400 Gbps IB: transfer = 0.17 ms, but AllToAll latency with 64 participants = ~2-5 ms (synchronization-dominated). Step time: 1 ms compute + 5 ms AllToAll = 6 ms → 83% communication overhead. Popularity-aware sharding: Replicate top 1% on each GPU: 10.2 GB per GPU (fits in 80 GB alongside 15.8 GB of cold shards). 50% of lookups are now local (0 ms communication). Remaining 50% still need AllToAll but with 50% less data: ~1-2.5 ms. Step time: 1 ms compute + 2.5 ms AllToAll = 3.5 ms → 71% utilization improvement. Memory cost: 10.2 GB × 64 GPUs = 653 GB total replication overhead — but GPU memory is cheaper than GPU idle time.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We serve a 13B model with static batching: we collect 32 requests, run them together, and return all results when the longest sequence finishes. Average latency is 4 seconds, but P99 is 12 seconds. The team wants to try vLLM's continuous batching. Explain the systems mechanism and quantify the improvement."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Continuous batching just means we use a bigger batch size, so throughput goes up." This misses the fundamental scheduling innovation — it's not about batch size, it's about batch dynamics.
Realistic Solution: In static batching, all requests in a batch must wait for the longest sequence to finish generating. If 31 requests finish in 2 seconds but one request generates a 500-token response taking 10 seconds, all 31 completed requests sit in memory waiting — wasting both GPU cycles (padding with no-ops) and user time. Continuous batching (also called iteration-level scheduling) operates at the granularity of individual decode steps. After each token generation step, the scheduler can: (1) evict completed requests immediately (freeing their KV-cache memory); (2) admit new requests from the queue into the freed slots. The GPU is never idle and never wasting cycles on completed requests. The result: short requests return immediately upon completion (latency improvement), and the freed memory slots are instantly reused for new requests (throughput improvement).
Napkin Math: Static batching (batch=32): Sequence lengths vary: 20% generate 50 tokens (~1 s), 60% generate 200 tokens (~4 s), 20% generate 500 tokens (~10 s). All 32 requests wait for the slowest: 10 s per batch. Throughput: 32 requests / 10 s = 3.2 req/s. GPU utilization: after 1 s, 6 requests are done but still occupying slots → 19% waste. After 4 s, 26 requests done → 81% waste. Average utilization: ~40%. Continuous batching: Short requests (50 tokens) complete in 1 s and exit immediately. Their slots are filled by new requests from the queue. Medium requests complete in 4 s. Long requests complete in 10 s. Average latency: $0.2 \times 1 + 0.6 \times 4 + 0.2 \times 10 = $ 4.6 s (vs 10 s static). But crucially, the GPU processes new requests in freed slots: effective throughput ≈ 32 slots × continuous fill ≈ 7-8 req/s — a 2-2.5× throughput improvement. P99 drops from 12 s to ~10 s (the inherent generation time), and P50 drops from 4 s to ~2 s.
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "We're building a new 2,048-GPU H100 training cluster. The network team proposes two topologies: a traditional fat-tree (Clos) network and NVIDIA's rail-optimized topology. The fat-tree costs $12M for switches and optics; the rail-optimized design costs $8M. The network team says 'rail-optimized saves 33% and NVIDIA recommends it.' Should we trust this recommendation, or is the fat-tree worth the premium?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "NVIDIA recommends rail-optimized, so it must be better for all workloads" or "Fat-tree has full bisection bandwidth, so it's always superior." Both ignore that the optimal topology depends on the traffic pattern, which depends on the parallelism strategy.
Realistic Solution: The two topologies optimize for different communication patterns. Fat-tree (Clos): Provides full bisection bandwidth — any GPU can communicate with any other GPU at line rate simultaneously. This is ideal for workloads with unpredictable or all-to-all communication patterns (e.g., expert parallelism in MoE models, where the router sends tokens to arbitrary GPUs). Cost: requires $O(N)$ spine switches, each with full-bandwidth uplinks. Rail-optimized: Groups GPUs into "rails" — GPU 0 from every node is on rail 0, GPU 1 on rail 1, etc. Each rail is a separate, smaller network. This is optimal for data-parallel training where AllReduce happens independently within each rail (GPU $i$ only communicates with GPU $i$ on other nodes). Cost: fewer switches, simpler cabling, 30-40% cheaper. The trade-off: rail-optimized has zero cross-rail bandwidth. If your parallelism strategy ever requires GPU 0 on node A to talk to GPU 3 on node B (e.g., pipeline parallelism, tensor parallelism across nodes, or MoE expert routing), traffic must hairpin through the node's internal NVSwitch, halving effective bandwidth and adding latency.
Napkin Math: 2,048 GPUs = 256 nodes × 8 GPUs/node. Fat-tree: 256 leaf switches (1 per node) + 128 spine switches. Each leaf: 8 × 400G downlinks (to GPUs) + 8 × 400G uplinks (to spines). Bisection bandwidth: $256 \times 8 \times 400\text{ Gbps} / 2 = $ 409.6 Tbps. Cost: 384 switches × ~$25K + optics ≈ $12M. Rail-optimized: 8 independent rail networks, each connecting 256 GPUs (one per node). Each rail: 16 leaf switches + 8 spine switches = 24 switches per rail × 8 rails = 192 switches. Per-rail bisection bandwidth: $256 \times 400\text{ Gbps} / 2 = $ 51.2 Tbps per rail. Cost: 192 switches × ~$25K + optics ≈ $8M. The tax: With pure data parallelism (DP=2048), rail-optimized is perfect — each GPU only talks to its rail peers. But with 3D parallelism (TP=8, PP=4, DP=64): TP is intra-node (NVLink, no network needed). PP requires node-to-node communication between different GPU indices (GPU 7 on node A → GPU 0 on node B) — this is cross-rail traffic. On fat-tree: direct path at 400 Gbps. On rail-optimized: must traverse NVSwitch within node A (GPU 7 → GPU 0) then rail 0's network — effective bandwidth drops to ~200 Gbps and adds ~2 μs latency per hop. For pipeline-parallel bubble overhead of 5%, this cross-rail penalty can push it to 8-12%.
📖 Deep Dive: Volume II: Compute Infrastructure
</details>Interviewer: "Your team runs a 13B model on A100 80 GB GPUs for long-running inference sessions. After ~6 hours of continuous serving, nvidia-smi shows VRAM usage has crept from 32 GB to 74 GB, and the next request triggers an OOM kill. Restarting the process fixes it for another 6 hours. The model weights haven't changed. What is leaking and how do you find it?"
Common Mistake: "The model must be loading duplicate weights over time" or "Just increase the GPU memory." The first is implausible for a static model; the second masks the root cause and fails on the next model size up.
Realistic Solution: The leak is almost certainly in the KV-cache allocator. Most serving frameworks pre-allocate a KV-cache pool, but when requests are cancelled mid-generation (client disconnects, timeouts), the allocated KV-cache blocks may not be returned to the free pool. Each orphaned block holds memory proportional to the sequence length and number of layers. Over thousands of cancelled requests, these orphaned blocks accumulate. A secondary source is CUDA graph capture: if the framework re-captures CUDA graphs for new input shapes (dynamic batching with varying sequence lengths), each captured graph allocates a private workspace that is never freed. Diagnosis: use torch.cuda.memory_stats() to track allocated_bytes.all.current vs reserved_bytes.all.current — a growing gap between reserved and allocated indicates fragmentation or leaks in the caching allocator. For the KV-cache specifically, instrument the block manager to log allocations and frees, then diff them.
Napkin Math: KV-cache per request (13B model, 32 layers, 40 heads, d=128, seq=2048, FP16): $2 \times 32 \times 40 \times 128 \times 2048 \times 2 = $ 1.34 GB. If 1% of requests are cancelled without freeing their KV blocks, and the server handles 600 req/hr: leaked blocks/hr = 6 × 1.34 GB = 8 GB/hr. After 6 hours: 48 GB leaked — exactly matching the observed creep from 32 GB to ~80 GB. Fix: implement a KV-cache garbage collector that scans for blocks with no active request reference every 60 seconds. CUDA graph leak: each captured graph workspace ≈ 50–200 MB. With 100 unique input shapes over 6 hours: up to 20 GB of dead graph workspaces.
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "You're training a 30B model across 64 H100 GPUs (8 nodes × 8 GPUs). At random intervals — sometimes after 2 hours, sometimes after 20 — the job hangs and eventually dies with NCCL WARN Timeout on rank 47. The hang always resolves to a different rank. Your network monitoring shows no packet loss. What's happening?"
Common Mistake: "It's a network issue — increase NCCL_TIMEOUT to give the slow node more time." Increasing the timeout just delays the crash; it doesn't fix the root cause. And the randomness across ranks rules out a single bad NIC.
Realistic Solution: The timeout means one GPU fell behind the collective — all other ranks are waiting for rank 47 to contribute its chunk in the AllReduce ring, but rank 47 is still computing. The randomness across ranks points to a stochastic slowdown, not a deterministic one. Top suspects: (1) GPU thermal throttling — one node's cooling is marginal; under sustained load, a random GPU hits 83°C and throttles from 700W to ~500W, falling behind the collective. The "random rank" pattern occurs because different GPUs throttle at different times depending on workload and airflow. (2) ECC error correction storms — intermittent HBM errors trigger ECC correction, which stalls memory accesses for microseconds. At 64 GPUs, even rare per-GPU events become frequent cluster-wide. (3) OS-level interference — a cron job, log rotation, or kernel memory compaction on the host CPU stalls the PCIe DMA engine, delaying the GPU's NCCL kernel launch. Diagnosis: set NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=COLL to log per-rank timing. Correlate the slow rank with nvidia-smi -q -d TEMPERATURE,ECC and host-level dmesg logs.
Napkin Math: AllReduce for 30B × 4 bytes = 120 GB across 64 GPUs via ring: bandwidth time = $2 \times 120 / 50 = $ 4.8 s. Each ring step must complete within a per-step budget of $4.8 / (2 \times 63) \approx $ 38 ms. A thermally throttled GPU running at 71% speed delays its compute by ~30%, adding ~11 ms per step. Over 126 ring steps, this accumulates to 1.4 s of straggler delay — within NCCL's default 5-minute timeout. But if the throttled GPU also hits an ECC storm (adding ~5 ms per correction, 10 corrections per step), total delay = $(11 + 50) \times 126 = $ 7.7 s per AllReduce. With 1000 AllReduces per epoch, the job falls progressively behind until the accumulated delay exceeds the timeout. Fix: monitor
nvidia-smi --query-gpu=clocks_throttle_reasons.hw_thermal_slowdownper GPU and proactively drain throttling nodes.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're 45 days into a 90-day pre-training run on 4,096 H100 GPUs. The loss curve looks smooth — no spikes, no NaNs. But your weekly eval on a 10k-sample benchmark shows accuracy plateaued 2 weeks ago and is now decreasing, diverging from the scaling law prediction by 6 points. Every software check passes. A hardware engineer suggests a cosmic ray bit flip corrupted a weight weeks ago. How is that even possible, and how do you find which of 70 billion parameters is wrong?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "ECC memory would catch any bit flip" or "A single corrupted weight out of 70 billion can't matter — SGD would just train past it." ECC catches most single-bit errors in HBM, but not errors in register files, SRAM caches, or during computation (Silent Data Corruption). And a single flipped bit in a high-magnitude weight can permanently bias an entire attention head.
Realistic Solution: Silent Data Corruption (SDC) bypasses ECC because the error occurs in logic (ALU, Tensor Core) or in unprotected SRAM, not in HBM. A bit flip in a BF16 weight's exponent bits can change a value from 0.5 to 128.0 (flipping bit 14 shifts the exponent by 7, multiplying the value by $2^7 = 128$). If this happens in a query/key projection weight, every token's attention distribution is corrupted, biasing the head toward or away from certain positions. The training loss may still decrease because the other 79 layers and 31 heads compensate, but the model's capability degrades — visible only on eval benchmarks that test specific reasoning. Finding the corrupted parameter: (1) compare the current checkpoint's weight statistics (per-layer mean, std, max) against the checkpoint from 2 weeks ago when eval was still on-track; (2) look for any single parameter whose magnitude is an outlier (>10σ from its layer's distribution); (3) use the scaling law prediction to estimate when the corruption occurred by finding the eval inflection point, then diff checkpoints around that date.
Napkin Math: BF16 weight = 16 bits. A flip in bit 14 (highest exponent bit): value changes by factor of $2^7 = 128\times$. A weight of 0.01 becomes 1.28. In a query projection matrix of shape [8192, 128], this one corrupted weight biases every token's query vector by +1.28 in one dimension. Attention logits shift by $1.28 \times K^T$ ≈ 0.5–2.0 nats — enough to redirect 30–60% of attention mass to wrong positions. At 4,096 GPUs with 80 GB HBM each: total HBM = 327 TB. Google/Meta report SDC rates of ~1–2 per 1,000 GPU-days. Over 45 days: expected SDC events = $4096 \times 45 / 1000 \times 1.5 \approx $ 276 events. Most corrupt activations (transient, overwritten next step). But ~1% corrupt weights (persistent) = ~3 weight corruptions. Detection: per-layer weight norm monitoring adds <0.01% overhead. Weekly checkpoint diffing: compare
torch.max(abs(ckpt_new - ckpt_old))per layer — a corrupted weight shows as a single outlier of 100× the normal per-step delta.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're fine-tuning a 7B model on 8 A100 GPUs. At step 12,400 the training loss suddenly spikes from 1.8 to 45.0, then gradually recovers over the next 200 steps. The spike happens at the exact same step every time you restart from the same checkpoint. What's in that batch, and how do you prove it?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The learning rate is too high — reduce it" or "This is normal training noise." The reproducibility at the exact same step rules out stochastic noise, and the spike magnitude (25×) is far beyond normal gradient variance.
Realistic Solution: The deterministic reproduction at step 12,400 means the data loader, with its fixed seed, serves a specific mini-batch at that step that contains pathological examples. Common culprits: (1) a corrupted sample with extremely long repetitive sequences that cause attention to produce near-uniform distributions, generating huge gradients in the softmax backward pass; (2) a mislabeled example where the target is nonsensical (e.g., a truncated UTF-8 sequence decoded as garbage tokens), producing a cross-entropy loss orders of magnitude above normal; (3) a duplicate of the prompt as the target, causing the model to receive contradictory supervision. Proof: log the batch indices at step 12,400, extract those samples, compute per-sample loss — the pathological sample will have loss >100× the batch mean. Fix: implement per-sample gradient clipping or loss clipping that caps individual sample contributions, and add a data quality filter that removes samples with anomalous token distributions.
Napkin Math: Normal per-sample cross-entropy loss ≈ 1.8 nats. A garbage-target sample where the model assigns ~0.001 probability to each target token: per-token loss = $-\ln(0.001) = 6.9$ nats. Over a 512-token sequence: sample loss = $6.9 \times 512 = $ 3,533 nats. In a batch of 32 samples: batch mean loss = $(31 \times 1.8 + 3533) / 32 = $ 112. With gradient scaling, the gradient norm from this one sample is ~$3533 / 1.8 = 1963\times$ normal. Without per-sample clipping, this single sample's gradient dominates the entire update, pushing all 7B parameters in a nonsensical direction. Recovery takes ~200 steps because the learning rate is small enough that 200 normal updates gradually undo the damage: $200 \times \text{lr} \times \text{normal_grad} \approx 1 \times \text{lr} \times \text{bad_grad}$.
📖 Deep Dive: Volume I: Model Training
</details>Interviewer: "Your Python-based inference server on an A100 serves a 7B model with P50 latency of 40 ms. But every ~30 seconds, a burst of requests hits P99 latency of 800 ms — 20× the normal. The GPU utilization trace shows brief drops to 0% that correlate exactly with the spikes. The model and batch size haven't changed. What's stealing 760 ms from your GPU?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The GPU is overheating and throttling" or "There's network congestion from other services." Thermal throttling is gradual (not a sharp 0% drop), and network issues wouldn't zero out GPU utilization.
Realistic Solution: This is a CPython garbage collector (GC) stop-the-world pause. Python's cyclic GC (triggered by gc.collect() or automatically when the generation-2 threshold is reached) freezes all Python threads while it traces object references. During this pause, no new CUDA kernels can be launched because the Python threads that submit them are frozen. The GPU drains its kernel queue in ~1–2 ms and then sits idle until the GC completes. With a large Python heap (serving frameworks maintain request queues, tokenizer caches, and KV-cache metadata as Python objects), a gen-2 collection can take 500–800 ms. The ~30-second interval matches the default gen-2 threshold: after 10 gen-1 collections (each triggered by ~700 allocations), a gen-2 sweep runs. Fix: (1) disable automatic GC (gc.disable()) and run gc.collect() explicitly during low-traffic windows; (2) reduce Python-side object churn by pre-allocating buffers; (3) move the hot path to C++/Rust (like vLLM's C++ scheduler) so the GC has fewer objects to trace.
Napkin Math: Python heap for a serving process: ~2–4 GB of Python objects (request metadata, tokenizer vocab, batch scheduler state). Gen-2 GC traces all objects: at ~5M objects, tracing at ~8M objects/s takes 625 ms. During this pause: GPU kernel queue depth ≈ 20 kernels × 0.05 ms each = drains in 1 ms. GPU sits idle for remaining 624 ms. Requests arriving during the pause queue up: at 100 req/s, ~62 requests are delayed. Each sees an additional 624 ms latency → P99 spike. After disabling auto-GC and running manual collection every 60 s during a scheduled 50 ms micro-pause (by reducing heap to <500K objects via C++ offload): GC time drops to 12 ms, P99 drops to 52 ms.
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "Your LLM serving endpoint supports 32k context. On Monday, your cluster starts OOM-killing pods every few minutes. The traffic volume hasn't increased, but you notice a handful of users sending prompts that are exactly 32,000 tokens — the maximum. Before Monday, average prompt length was 500 tokens. How does a 64× increase in prompt length from a few users crash the entire cluster, and what's your emergency response?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just reject prompts over a certain length" or "Add more GPUs to handle the load." Rejecting long prompts breaks legitimate use cases, and adding GPUs doesn't help if the scheduling algorithm is the problem.
Realistic Solution: The KV-cache memory for a single request scales linearly with sequence length. A 32k-token request consumes 64× the KV-cache of a 500-token request. If the scheduler admits requests based on count (e.g., "batch up to 32 requests") rather than memory, a few 32k requests can exhaust the entire KV-cache budget that normally serves hundreds of short requests. This is effectively a resource exhaustion attack — intentional or not. The cluster crashes because: (1) the scheduler admits 32 requests including several 32k-token ones; (2) KV-cache allocation exceeds VRAM; (3) the CUDA OOM kills the serving process; (4) Kubernetes restarts the pod, which immediately admits the same queued requests and crashes again (crash loop). Emergency response: (1) implement admission control based on memory budget, not request count — estimate KV-cache cost before admitting: $\text{KV_bytes} = 2 \times L \times H \times d_h \times S \times 2$; (2) set per-user rate limits on total token throughput, not just request count; (3) implement preemption — if a new request would cause OOM, preempt (pause and swap to CPU) the lowest-priority in-flight request.
Napkin Math: 7B model on A100 80 GB. Weights = 14 GB. Free for KV-cache = 66 GB. KV-cache per token per request (32 layers, 32 heads, d=128, FP16): $2 \times 32 \times 32 \times 128 \times 2 = $ 524 KB/token. At 500-token avg: 262 MB/request. Batch of 32 normal requests: $32 \times 262\text{ MB} = $ 8.4 GB — fits easily. At 32k tokens: 16.8 GB/request. Just 4 adversarial requests: $4 \times 16.8 = $ 67.2 GB — exceeds the 66 GB budget → OOM. Those 4 requests consume more memory than 256 normal requests. Memory-aware scheduling: admit requests until $\sum_i \text{KV}(S_i) \leq 66\text{ GB}$. This limits concurrent 32k requests to 3, while allowing 250+ concurrent 500-token requests. PagedAttention (vLLM) helps by allocating KV-cache in 4 KB pages on-demand rather than pre-allocating for max length, reducing waste from requests that don't use their full context window.
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "After upgrading from CUDA 11.8 to CUDA 12.2, your 13B model's accuracy on your internal benchmark drops by 2.3 points. The model weights are identical — same checkpoint file. The training team swears nothing changed. How can the same weights produce different accuracy with a different CUDA version?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "It must be a bug in CUDA 12.2 — file a bug with NVIDIA" or "2.3 points is within noise." A 2.3-point drop on a stable benchmark with identical weights is statistically significant and reproducible, but it's not a bug — it's a consequence of how floating-point math works.
Realistic Solution: CUDA version upgrades change the cuDNN and cuBLAS kernel implementations. Different kernel implementations use different reduction orders, tiling strategies, and fused operations — all of which change the order of floating-point additions. Due to floating-point non-associativity ($(a + b) + c \neq a + (b + c)$ in FP16/BF16), different reduction orders produce different results at the bit level. For a single operation, the difference is in the last mantissa bit (~0.1% relative error). But in a 32-layer Transformer, these differences compound through every matmul, softmax, and LayerNorm. By the output layer, accumulated rounding differences can shift logits by 0.01–0.1, which is enough to change the argmax token for ~2–5% of predictions. The fix: (1) set torch.backends.cudnn.deterministic = True and torch.use_deterministic_algorithms(True) — this forces deterministic kernels at a 5–15% performance cost; (2) if determinism is too expensive, re-validate the model on your benchmark after every CUDA upgrade and establish a regression threshold; (3) for production, pin CUDA versions in your container images and only upgrade with a full eval cycle.
Napkin Math: BF16 mantissa = 7 bits → relative precision = $2^{-7} \approx 0.78%$. A single matmul accumulating 4096 products: worst-case rounding error ≈ $\sqrt{4096} \times 2^{-7} \approx 50%$ relative — but in practice, errors are random and partially cancel, giving ~$0.78% \times \sqrt{4096 / 3} \approx 29%$... no, the actual measured per-layer divergence is ~0.01–0.1% because cuBLAS accumulates in FP32 internally. Over 32 layers with residual connections: divergence compounds as $\epsilon_{\text{total}} \approx 32 \times 0.05% \approx 1.6%$ relative shift in output logits. On a 50k-vocab softmax, a 1.6% logit shift changes the top-1 prediction for tokens where the top-2 logit gap is <0.1 — roughly 2–5% of tokens, matching the 2.3-point accuracy drop.
📖 Deep Dive: Volume I: Neural Computation
</details>Interviewer: "You're training a 70B model on 256 GPUs. At step 80,000 a node fails. You restart from the step-79,000 checkpoint on NFS. The model loads without errors, but training loss immediately jumps to 11.0 (vs 2.1 before the crash) and never recovers. The step-78,000 checkpoint works fine. What happened to the step-79,000 checkpoint?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The checkpoint file is corrupted on disk — use a different storage backend." This is half right (the checkpoint is corrupted) but misses why and how to prevent it.
Realistic Solution: NFS has weak consistency guarantees under concurrent writes. During checkpointing, all 256 GPUs (32 nodes × 8 GPUs) write their shards to NFS simultaneously. If two nodes' writes target overlapping NFS blocks (common with ZeRO-3 where optimizer states are gathered and written by the coordinator), NFS's close-to-open consistency model means a reader may see a partially written file — some blocks from the new checkpoint, some from the previous one, or some filled with zeros. The step-79,000 checkpoint is a Frankenstein: the first 60% of the optimizer state is from step 79,000, but the last 40% is stale data from step 78,000 (or zeros from an incomplete write). The model loads because the tensor shapes are correct, but the optimizer state is inconsistent — Adam's momentum and variance estimates are from two different points in training, causing the first update to produce a catastrophically wrong step. Fix: (1) write checkpoints to a temporary path, then atomically rename (os.rename is atomic on POSIX); (2) compute and verify SHA-256 checksums of each shard; (3) use a two-phase commit: all ranks write, all ranks verify checksums, then the coordinator writes a .complete sentinel file — only checkpoints with the sentinel are valid for restart.
Napkin Math: 70B model checkpoint with ZeRO-3: total state = 70B × 16 bytes = 1.12 TB. 32 nodes writing simultaneously to NFS at ~2 GB/s per node aggregate: write time = $1120 / (32 \times 2) = $ 17.5 seconds. NFS block size = 1 MB. Total blocks = $1.12 \times 10^{12} / 10^6 = $ 1.12 million blocks. If the coordinator node crashes at second 14 (80% complete): ~224,000 blocks are missing or stale. The corrupted optimizer state means Adam's $m_t$ and $v_t$ are mismatched: $m_t$ from step 79,000 but $v_t$ from step 78,000 for 40% of parameters. The first update computes $\theta_{t+1} = \theta_t - \text{lr} \times m_t / (\sqrt{v_{t-1000}} + \epsilon)$ — the denominator is wrong by up to 1000 steps of variance accumulation, producing updates that are 2–10× too large or too small. Checksum verification adds <1% overhead: SHA-256 of 1.12 TB at 2 GB/s = 560 seconds across 32 nodes in parallel = 17.5 s — doubling checkpoint time but preventing a $500K training run from being wasted.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "Every June through August, your 512-GPU H100 training cluster in Dallas, Texas runs 25% slower than in winter. The ops team has verified: same code, same model, same batch size. nvidia-smi shows GPU clocks dropping from 1980 MHz to 1410 MHz during afternoon hours. What's the physical root cause, and what's the cheapest fix?"
Common Mistake: "The GPUs must be defective — RMA them" or "Upgrade the air conditioning." The GPUs are working exactly as designed, and upgrading HVAC for peak summer may be cost-prohibitive.
Realistic Solution: This is thermal throttling driven by ambient temperature. Dallas summer afternoons hit 40°C+ (104°F+). The data center's air-cooled CRAC units reject heat to outdoor air via condenser coils. When outdoor temperature rises, the temperature delta between the coolant loop and ambient shrinks, reducing cooling capacity. The server inlet air temperature rises from the design point of 20°C to 30–35°C. H100 GPUs throttle when junction temperature exceeds 83°C: the firmware reduces clock speed and power draw to stay within thermal limits. At 1410 MHz vs 1980 MHz, the GPU delivers $1410/1980 = 71%$ of peak performance — matching the 25% slowdown. The cheapest fix isn't more cooling — it's time-shifting workloads. Schedule heavy training jobs from 8 PM to 10 AM when ambient drops to 25°C. During peak afternoon hours, run lighter workloads (evaluation, data preprocessing, checkpoint management). This costs $0 in hardware and recovers most of the lost throughput.
Napkin Math: H100 TDP = 700W at 1980 MHz. Throttled to 1410 MHz: power ≈ $700 \times (1410/1980)^2 \approx $ 356W (power scales roughly with $V^2 f$, and voltage drops with frequency). Heat rejection needed: 512 GPUs × 700W = 358 kW at full speed. CRAC capacity at 20°C ambient: ~400 kW. CRAC capacity at 40°C ambient: ~280 kW (30% reduction from halved temperature delta). At 280 kW cooling capacity, GPUs must throttle to $280/512 = $ 547W average — but 547W still exceeds the 356W throttle point, so the actual throttle is set by per-GPU junction temperature, not aggregate cooling. Time-shifting: Dallas drops below 27°C by 9 PM (May–Sep). Training at night: 12 hours × full speed = 12 effective hours. Training during day: 12 hours × 75% speed = 9 effective hours. Total = 21 effective hours/day vs 18 hours/day (all-day throttled). Alternatively, rear-door liquid cooling retrofit at ~$500/GPU = $256K eliminates ambient dependency entirely — payback in ~2 months of recovered GPU-hours at $3.50/GPU-hr.
📖 Deep Dive: Volume II: Compute Infrastructure
</details>Interviewer: "Your 256-GPU training job stalls for 30–90 seconds every 10–20 minutes, then resumes at full speed. NCCL_DEBUG logs show no timeouts — the collectives complete, just slowly. Your IB switch logs show a port on leaf switch 7 toggling UP/DOWN every 12 minutes. The ops team says 'it's just one port — it only affects one GPU.' Why does one flapping link stall 256 GPUs?"
Common Mistake: "One bad link only affects the one GPU connected to it — the other 255 should be fine" or "InfiniBand has redundant paths, so a single link failure is handled automatically." Both underestimate how collective operations create global dependencies.
Realistic Solution: In a ring or tree AllReduce, every GPU is a link in a chain. The collective cannot complete until every participant contributes its data. When the IB link flaps (goes DOWN), the Subnet Manager (SM) must: (1) detect the failure (~500 ms); (2) recalculate routing tables for the entire fabric (~1–5 s for 256 endpoints); (3) distribute new forwarding tables to all switches (~2–5 s); (4) the affected GPU's NCCL connection times out and retries (~5–30 s depending on NCCL_IB_TIMEOUT). During this entire sequence, all 255 other GPUs are blocked in the collective, waiting for the one GPU on the flapping port. When the link comes back UP, the SM recalculates routes again. The 12-minute flap cycle means the fabric is constantly reconverging, and each reconvergence stalls the entire training job. One flapping link is worse than a permanently dead link (which would be routed around once and forgotten).
Napkin Math: SM reconvergence time for 256-node fabric: 3–8 seconds. NCCL retry with backoff: 5–30 seconds. Total stall per flap event: 10–40 seconds (matching observed 30–90 s with variance). Flap interval: 12 minutes. Training step time: ~4 seconds. Steps lost per flap: $30 / 4 \approx $ 8 steps. Flaps per hour: 5. Steps lost per hour: 40 steps. Over 24 hours: 960 steps lost. At 256 GPUs × $3.50/GPU-hr: cost of idle GPUs during stalls = $256 \times 3.50 \times (960 \times 4 / 3600) \approx $ $956/day. Fix: replace the flapping cable/transceiver ($200) or configure the SM with a link flap dampening policy that holds a port DOWN for 5 minutes after 3 flaps in 10 minutes, forcing NCCL to route around it permanently. The $200 cable fix saves $956/day.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "Your recommendation system has a 500 GB embedding table sharded across 32 A100 80 GB GPUs. During a flash sale event, GPU 14 OOMs while all other GPUs sit at 40% memory utilization. The embedding table hasn't changed size. What happened on GPU 14?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "GPU 14 has a hardware defect — swap it out" or "The sharding is uneven — rebalance the table." The sharding is perfectly even by entry count, and the hardware is fine.
Realistic Solution: The flash sale created a massive traffic spike for a small set of product IDs. With hash-based sharding (embedding_id % 32), the hot product embeddings happen to cluster on GPU 14's shard. During the sale, GPU 14 receives 50× more lookup requests than average, and each lookup's backward pass accumulates gradients for those hot entries. The gradient accumulation buffer for frequently accessed entries grows because the optimizer step hasn't run yet (it's waiting for the AllReduce to complete, which is waiting for all GPUs). Meanwhile, the forward pass continues allocating activation memory for the flood of requests routed to GPU 14. The combination of gradient accumulation + activation memory for the hot-shard traffic exceeds GPU 14's 80 GB budget. The other 31 GPUs are fine because their entries aren't being accessed. Fix: (1) use consistent hashing with virtual nodes to spread hot entries across multiple GPUs; (2) replicate the top-K hottest embeddings (identified from access logs) on all GPUs; (3) implement a memory-aware request router that sheds load from GPUs approaching their memory limit.
Napkin Math: 500 GB table / 32 GPUs = 15.6 GB/GPU for embeddings. Remaining per-GPU budget: 80 - 15.6 = 64.4 GB. Normal traffic: each GPU handles ~3% of lookups (roughly uniform). GPU 14 during flash sale: handles 30% of all lookups (hot products hash to shard 14). If total batch = 65,536 lookups: GPU 14 gets ~19,660 lookups. Each lookup's gradient: 128 dims × 4 bytes (FP32 grad) = 512 bytes. Gradient buffer for 19,660 unique hot entries: $19660 \times 512 = $ 10 MB — small. But the activation memory for processing 19,660 lookups through the interaction network: ~2 KB per lookup × 19,660 = 38 MB per micro-batch. With 100 micro-batches queued during the AllReduce stall: $100 \times 38\text{ MB} = $ 3.8 GB of queued activations. The real killer: the optimizer's gradient history (Adam m, v) for hot entries gets updated every step, but the memory for cold entries is also reserved: $500\text{ GB} \times 12 / 32 = $ 187 GB of optimizer state per GPU — far exceeds 80 GB. The actual deployment must use CPU-offloaded optimizer state, and the OOM occurs when the hot-entry gradient traffic exceeds the GPU↔CPU PCIe bandwidth budget.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "Friday evening, the ops team updates the NVIDIA driver from 535.104 to 545.23 across your serving fleet for a security patch. Monday morning, all TensorRT inference engines fail to load with INVALID_CONFIG errors. The model files haven't changed. Rolling back the driver fixes it. How can a driver update break a model file, and what's the correct deployment practice?"
Common Mistake: "TensorRT models are hardware-independent — a driver update shouldn't affect them" or "Just rebuild the TensorRT engines on the new driver." The first is wrong; the second is the fix but misses why and how to prevent recurrence.
Realistic Solution: TensorRT engines are not portable across driver versions. When you build a TensorRT engine, the builder selects specific kernel implementations (called "tactics") from cuDNN and cuBLAS based on the current GPU architecture and the installed library versions. These tactics are compiled into the serialized engine file as binary GPU code (SASS). A driver update changes the cuDNN/cuBLAS libraries bundled with the driver, which may deprecate or rename tactics that the engine references. When the engine tries to load a tactic that no longer exists in the new driver's library, it fails with INVALID_CONFIG. This is by design — TensorRT trades portability for performance by baking in hardware-specific optimizations. The correct practice: (1) pin the driver version in your container image, not the host; (2) store TensorRT engines with metadata (driver version, CUDA version, GPU architecture) and rebuild automatically when any component changes; (3) maintain a CI pipeline that rebuilds engines on driver update and validates accuracy before fleet rollout.
Napkin Math: TensorRT engine build time for a 13B model: 15–45 minutes (profiling thousands of tactic combinations). Engine file size: ~28 GB (2× the FP16 weights due to embedded tactic metadata and workspace allocations). Rebuilding across a 100-GPU fleet: if engines are built per-GPU, that's 100 × 30 min = 50 GPU-hours of downtime. With a centralized build (one engine per GPU architecture, copied to all): 30 min build + 5 min distribution = 35 minutes total downtime. The Friday driver update without engine rebuild caused ~60 hours of serving downtime (Friday 6 PM to Monday 8 AM). At $0.50/request and 1000 req/s: lost revenue = $0.50 × 1000 × 60 × 3600 = $108M — a catastrophic incident from a "routine" update. Prevention: driver updates go through a staging environment that automatically rebuilds TensorRT engines and runs accuracy validation before promoting to production.
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "Your image classification model (ResNet-50 on A100) was deployed 6 months ago with 94% accuracy. Without any model updates, accuracy has gradually degraded to 87%. The model weights are identical to deployment day — you verified the checkpoint hash. The training team says the model is fine. What's silently changing?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model must have been accidentally updated" or "This is just random variance in the test set." The checkpoint hash verification rules out the first, and a 7-point drop over 6 months is a systematic trend, not noise.
Realistic Solution: This is data distribution drift interacting with Batch Normalization's frozen statistics. During training, BatchNorm layers compute running mean ($\mu$) and running variance ($\sigma^2$) from the training data. At inference time, these frozen statistics are used to normalize inputs: $\hat{x} = (x - \mu_{\text{train}}) / \sigma_{\text{train}}$. If the production data distribution shifts over time (lighting conditions change seasonally, camera firmware updates alter preprocessing, new product categories are added), the true mean and variance diverge from the frozen statistics. The normalization now distorts the data instead of normalizing it — pushing activations into regions the downstream layers never saw during training. The gradual 7-point drop over 6 months matches a slow seasonal drift (e.g., winter lighting vs summer lighting for a retail image classifier). Fix: (1) implement periodic BatchNorm recalibration — run a forward pass over recent production data with BatchNorm in training mode to update running statistics, then freeze again; (2) replace BatchNorm with LayerNorm or GroupNorm, which compute statistics per-sample and are immune to distribution drift; (3) deploy a data drift monitor that tracks input feature statistics and alerts when they diverge from training distribution.
Napkin Math: ResNet-50 has 53 BatchNorm layers. Training data mean pixel value: $\mu_{\text{train}} = 0.485$ (ImageNet). After 6 months of drift (new camera firmware increases brightness): $\mu_{\text{prod}} = 0.52$. Shift = $0.035$. BatchNorm normalizes: $\hat{x} = (0.52 - 0.485) / 0.229 = 0.153$ — the network sees a constant bias of 0.153 added to every activation. Through 53 layers with ReLU, this bias compounds: early layers clip negative activations that should have been positive (or vice versa), changing ~3–8% of activation signs per layer. By the final layer, the cumulative effect shifts the feature representation enough to flip predictions for 7% of samples — matching the 94% → 87% accuracy drop. BatchNorm recalibration: forward pass of 10,000 production images through ResNet-50 on A100: $10000 \times 8\text{ GFLOPs} = 80\text{ TFLOPs}$. At 989 TFLOPS: 0.08 seconds. Running this weekly costs essentially nothing and prevents the drift.
📖 Deep Dive: Volume I: Model Training
</details>Interviewer: "Your team fine-tuned a 7B model and deployed it to production. The weights are identical, but the serving cluster is suddenly OOMing under load, and the KV-cache is filling up 30% faster than expected for the exact same text inputs. Why does a tokenizer version mismatch cause a massive GPU memory leak, and how does it destroy your serving economics?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The serving framework must be allocating memory differently" or "There's a memory leak in the Python code." Both ignore the relationship between text, tokens, and hardware memory.
Realistic Solution: The serving pipeline is using an older or different tokenizer than training. If the training pipeline used tokenizer_v2 (which maps "machine learning" → tokens [4521, 3892], 2 tokens) but the serving pipeline loads tokenizer_v1 (which maps "machine learning" → [4521, 12, 3892, 44], 4 tokens), the exact same input text produces a longer sequence of token IDs. Because KV-cache memory scales linearly with sequence length, a less efficient tokenizer artificially inflates the memory footprint of every request. The model isn't just producing worse answers; it's physically consuming more VRAM per word of text, causing the batch scheduler to OOM or reject requests much earlier than expected.
Napkin Math: Typical tokenizer vocabulary: 32,000 tokens. If
v1is a naive BPE andv2is highly optimized for your domain,v1might average 1.5 tokens per word whilev2averages 1.1 tokens per word. For a 1,000-word prompt:v2= 1,100 tokens.v1= 1,500 tokens. That's a 36% increase in sequence length. For a 7B model (KV-cache = ~1MB per token), the prompt takes 1.1GB with the correct tokenizer, but 1.5GB with the mismatched one. Across a batch of 64 requests, you're wasting 25GB of HBM just storing the extra intermediate states of inefficiently tokenized text. This drops your maximum concurrent requests by 30%, forcing you to scale out the GPU cluster to handle the same QPS.
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "Your 128-GPU H100 cluster has been running for 14 months. You notice that one specific node (GPUs 40–47) consistently produces slightly different AllReduce results than other nodes — the gradient checksums diverge by 1–2 ULP (units in the last place) every ~100 steps. Training still converges, but your reproducibility tests fail. nvidia-smi shows no errors. What's degrading, and when does 'slightly different' become 'dangerously wrong'?"
Common Mistake: "1–2 ULP difference is just floating-point non-determinism — it's normal" or "If ECC reports no errors, the memory is fine." ULP-level differences are normal for different code paths, but the same code on the same hardware should produce bit-identical results. And ECC only corrects single-bit errors — it doesn't report correctable errors to the application by default.
Realistic Solution: The HBM on GPUs 40–47 is experiencing an elevated rate of correctable ECC errors (single-bit flips that ECC silently fixes). While each individual correction is invisible to the application, the correction process stalls the memory controller for ~100 ns per event. If a memory page has a marginal cell that flips frequently, the GPU's memory controller spends increasing time on corrections, subtly changing the timing of memory accesses. This timing change affects the order of floating-point reductions in parallel operations (warp-level reductions are timing-dependent when using non-deterministic atomics), producing the 1–2 ULP divergence. The danger: correctable ECC errors are precursors to uncorrectable errors (double-bit flips). HBM cells degrade over time — a cell that flips once per hour today may flip once per minute in 3 months, and eventually produce double-bit errors that ECC cannot fix, causing Silent Data Corruption. Check nvidia-smi -q -d ECC for the Volatile ECC Errors: Single Bit counter — if it's climbing, the HBM is degrading and the GPU should be proactively replaced.
Napkin Math: H100 HBM3: 80 GB across 6 stacks, each with billions of cells. Normal correctable ECC rate: <1 error per GPU per day. Degrading HBM: 100+ errors per GPU per day on the affected page. Each correction: ~100 ns stall. At 100 errors/day: total stall = 10 μs/day — negligible for performance. But the timing perturbation during a warp-level reduction (32 threads reducing 32 values) changes the addition order when one thread's memory access is delayed by 100 ns. FP32 addition: $(a + b) + c$ vs $a + (b + c)$ can differ by 1 ULP when $|a| \gg |c|$. Over 100 steps, ~1% of reductions are perturbed → 1–2 ULP divergence in the final AllReduce result. Failure progression: Correctable errors doubling every 2 months (typical HBM degradation curve). Month 14: 100/day. Month 16: 400/day. Month 18: 1,600/day. Month 20: first uncorrectable (double-bit) error → silent data corruption in a weight tensor. Proactive replacement at month 14 costs 1 GPU ($30K). Reactive replacement after SDC corrupts a training run costs the entire run ($500K+).
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You have 8 replicas of a 7B model behind an L7 load balancer, each on its own A100. Monitoring shows total QPS is normal, but P99 latency spiked from 80 ms to 2.4 seconds. Seven replicas report healthy latency. One replica's average latency is 18 seconds. The load balancer is configured for round-robin. Why is one slow replica destroying the P99 for the entire fleet?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Round-robin distributes traffic evenly, so the slow replica only affects 1/8 of requests — P99 should only increase slightly." This misunderstands how percentiles work with heterogeneous backends.
Realistic Solution: Round-robin sends exactly 1/8 of all requests to the slow replica. P99 means 1% of requests exceed the threshold. If 12.5% of requests (those hitting the slow replica) have 18-second latency, then all of those requests are in the worst 12.5% — far exceeding the 1% P99 threshold. The P99 is entirely determined by the slow replica. Worse, the slow replica creates cascading damage: (1) clients waiting 18 seconds for the slow replica hold open HTTP connections, consuming connection pool slots on the load balancer; (2) if clients have a 5-second timeout and retry, the retried request has a 1/8 chance of hitting the slow replica again, amplifying load; (3) the slow replica's request queue grows, making each subsequent request even slower (queuing theory: latency → ∞ as utilization → 1). Fix: (1) switch from round-robin to least-connections or latency-weighted routing — the slow replica naturally receives fewer requests; (2) implement health-check-based ejection: if a replica's P50 exceeds 2× the fleet median, remove it from the pool; (3) add client-side hedging: send the request to 2 replicas simultaneously and take the first response.
Napkin Math: 8 replicas, round-robin, total 1000 QPS → 125 QPS per replica. 7 healthy replicas: latency ~40 ms (P50), ~80 ms (P99). 1 slow replica: latency ~18 s. P99 calculation: Sort all 1000 requests by latency. The slowest 1% = 10 requests. The slow replica handles 125 requests — all 125 are slower than any healthy replica's request. P99 = the 10th-slowest request, which is from the slow replica: ~18 s. Even P87.5 is terrible: the 125th-slowest request is still from the slow replica. With least-connections routing: The slow replica accumulates connections (each held for 18 s). At steady state: slow replica has $125 \times 18 = 2250$ open connections. Healthy replicas have $125 \times 0.04 = 5$ each. Least-connections routes new requests to healthy replicas (5 connections) instead of the slow one (2250). The slow replica effectively drains to ~1 QPS. P99 drops back to ~80 ms because <0.1% of traffic hits the slow replica.
📖 Deep Dive: Volume I: ML Operations
</details>Interviewer: "Your Kubernetes cluster auto-scales model serving pods based on QPS. When traffic spikes, new pods are created, but the first requests to each new pod take 45 seconds instead of the normal 40 ms. Users see timeouts. The pod's readiness probe passes after 30 seconds. What's taking so long, and how do you get that 45 seconds down to under 5?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model is being downloaded from S3 — use a faster network" or "Pre-pull the container image." Both address parts of the problem but miss the dominant cost.
Realistic Solution: The 45-second cold start has four sequential phases: (1) Container pull (~5 s if image is cached on the node, ~30 s if not — the container image with CUDA runtime and framework is 15–25 GB); (2) Model weight loading (~10–15 s — reading 14 GB of FP16 weights from network storage into CPU RAM); (3) GPU transfer (~3–5 s — copying 14 GB from CPU RAM to GPU VRAM over PCIe Gen4 at ~25 GB/s effective); (4) CUDA/cuDNN warmup (~5–10 s — first inference triggers JIT compilation of CUDA kernels and cuDNN autotuning). The readiness probe passes after phase 3 (model is on GPU), but phase 4 means the first real request still pays the warmup penalty. Fixes: (1) pre-cache container images on all nodes using a DaemonSet; (2) store model weights on local NVMe (7 GB/s read) instead of network storage (1 GB/s); (3) use CUDA graph capture during startup to pre-compile kernels; (4) send synthetic warmup requests before marking the pod as ready. Combined: cold start drops from 45 s to ~4 s (0 s image pull + 2 s NVMe load + 0.6 s GPU transfer + 1.5 s warmup).
Napkin Math: Current breakdown: Image pull (cached): 5 s. Weight load from NFS (14 GB at 1 GB/s): 14 s. CPU→GPU transfer (14 GB at 25 GB/s PCIe): 0.56 s. CUDA warmup (kernel JIT + cuDNN autotune): 8 s. Readiness probe delay: 2 s. Total: ~30 s to ready + 15 s first-request warmup = 45 s effective. Optimized: Image pre-cached: 0 s. Weight load from local NVMe (14 GB at 7 GB/s): 2 s. GPU transfer: 0.56 s. Pre-compiled CUDA graphs (loaded from cache): 1.5 s. Warmup requests (3 synthetic): 0.12 s. Total: ~4.2 s. At 100 scale-up events/day, saving 41 s each: 4,100 s of reduced user-facing latency. More importantly: no more timeouts during traffic spikes.
📖 Deep Dive: Volume I: ML Operations
</details>Interviewer: "You're training a 13B model in mixed-precision (FP16 forward/backward, FP32 optimizer) on 32 H100 GPUs. At step 35,000, you start seeing Inf values in the loss, but only on 3 of 32 GPUs. The other 29 GPUs report normal loss values. After the AllReduce, all GPUs have Inf loss. What's happening on those 3 GPUs, and why does it infect the entire cluster?"
Common Mistake: "The 3 GPUs have hardware errors — replace them" or "Reduce the learning rate globally." The first is unlikely (3 simultaneous failures), and the second treats the symptom without understanding the data-dependent root cause.
Realistic Solution: The 3 GPUs received data-parallel micro-batches that happen to contain outlier examples — sequences with unusual token distributions that produce large activation magnitudes. In FP16, the maximum representable value is 65,504. When a large activation (e.g., from a rare token embedding with magnitude 200) passes through multiple layers with residual connections, values compound: $200 \times 1.5^{32} \approx 200 \times 1,262,177 \approx 2.5 \times 10^8$ — far exceeding FP16 max, producing Inf. The AllReduce averages gradients across all 32 GPUs: $(\text{normal} + \text{normal} + \ldots + \text{Inf}) / 32 = \text{Inf}$. One Inf in any GPU's gradient poisons the entire AllReduce, corrupting the update for all 32 GPUs. This is why dynamic loss scaling exists: it detects Inf in the gradients, skips the optimizer step, and halves the loss scale factor. But if the loss scaler's initial scale is too high, or if the outlier examples are frequent enough, the scaler enters a death spiral of repeated skips.
Napkin Math: FP16 max = 65,504. Typical activation magnitude after LayerNorm: ~1.0. Residual connection growth factor per layer: ~1.02× (small but compounds). After 80 layers: $1.0 \times 1.02^{80} = 4.88$ — safe. But an outlier embedding with magnitude 10.0: $10.0 \times 1.02^{80} = 48.8$ — still safe. With loss scaling factor of 1024 (typical initial value): gradient magnitudes = $48.8 \times 1024 = 49,971$ — dangerously close to FP16 max. A slightly larger outlier (magnitude 15): $15 \times 1.02^{80} \times 1024 = 74,957$ → overflow to Inf. The 3 GPUs' micro-batches contained such outliers. Fix: (1) cap loss scale at 512 instead of 1024; (2) implement per-sample gradient clipping before AllReduce; (3) add activation clamping after residual connections:
x = torch.clamp(x, -65000, 65000)— costs <0.1% compute.
📖 Deep Dive: Volume I: Neural Computation
</details>Interviewer: "Your training job on 8 A100 GPUs hangs at step 1 and never progresses. GPU utilization is 0%. CPU utilization is 100% across all cores. htop shows 128 Python processes in D (uninterruptible sleep) state. You set num_workers=16 per GPU in the DataLoader. What happened, and what's the maximum safe value for num_workers?"
Common Mistake: "More workers means faster data loading — set it as high as possible" or "The dataset is too large to fit in memory." The first ignores system resource limits, and the second is unrelated to the deadlock.
Realistic Solution: You've created 128 worker processes (16 per GPU × 8 GPUs) that are all competing for shared resources and deadlocking. The D state means they're waiting on I/O or kernel locks. Three mechanisms cause this: (1) Shared memory exhaustion — each DataLoader worker uses shared memory (/dev/shm) to pass tensors to the main process. Default /dev/shm in Docker is 64 MB. With 128 workers each trying to write a batch (e.g., 32 images × 224×224×3 × 4 bytes = 19 MB per batch): total demand = $128 \times 19\text{ MB} = 2.4\text{ GB}$ — far exceeding 64 MB. Workers block on shm_open(). (2) File descriptor exhaustion — each worker opens dataset files. At 128 workers with 10 FDs each: 1,280 FDs, potentially exceeding the process limit (default 1024). (3) CPU oversubscription — 128 CPU-bound workers on a 64-core machine means 2× oversubscription, causing context-switch thrashing. Fix: set num_workers = min(cpu_cores / num_gpus, 4) as a starting point. For 64 cores and 8 GPUs: num_workers=8. Increase /dev/shm to 16 GB in Docker (--shm-size=16g).
Napkin Math: Server: 64 CPU cores, 512 GB RAM, 8 GPUs.
num_workers=16per GPU × 8 GPUs = 128 workers. CPU oversubscription: $128 / 64 = 2\times$ — each worker gets only 50% of a core, and context-switching overhead wastes ~30% of that. Effective CPU per worker: 35%. Shared memory: 128 workers × 19 MB/batch = 2.4 GB needed. Docker default/dev/shm= 64 MB → 37× oversubscribed → deadlock. File descriptors: 128 × 10 = 1,280 FDs.ulimit -ndefault = 1024 → exceeded → workers fail onopen(). Safe configuration:num_workers=4per GPU × 8 GPUs = 32 workers. CPU utilization: $32/64 = 50%$ — leaves headroom for the main process and system tasks. Shared memory: $32 \times 19 = 608\text{ MB}$ — set--shm-size=2gfor safety. Data loading throughput at 4 workers: ~3,200 images/s per GPU (sufficient for most training at batch_size=32 with augmentation).
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "Your LLM serving endpoint on an H100 should deliver 40 tokens/s for a 13B model (bandwidth-bound: 26 GB weights / 3.35 TB/s ≈ 7.8 ms/token → 128 tokens/s theoretical). But you're measuring only 15 tokens/s. nvidia-smi shows GPU utilization flickering between 0% and 95% in a sawtooth pattern. CPU utilization is pegged at 100% on one core. Where's the bottleneck?"
Common Mistake: "The model is too large for the GPU — we need tensor parallelism" or "The batch size is too small." The model fits fine (26 GB on 80 GB), and the sawtooth GPU pattern reveals the real issue.
Realistic Solution: The sawtooth pattern (0% → 95% → 0% → 95%) means the GPU is alternating between waiting for input and processing it. The CPU is the bottleneck — specifically, the tokenizer and/or detokenizer running in Python on a single core. The Python GIL ensures only one thread executes Python bytecode at a time. The serving pipeline is: (1) CPU tokenizes input (Python, single-threaded) → (2) GPU runs model forward pass → (3) CPU detokenizes output token (Python, single-threaded) → (4) CPU samples next token → repeat. If tokenization + detokenization + sampling takes 50 ms on CPU, but the GPU forward pass takes only 7.8 ms, the GPU is idle for $50 / (50 + 7.8) = 87%$ of the time. Effective throughput: $1000 / (50 + 7.8) = 17.3$ tokens/s — matching the observed 15 tokens/s (with overhead). Fix: (1) use a Rust/C++ tokenizer (HuggingFace tokenizers library) that's 10–50× faster than pure Python; (2) batch tokenization — tokenize multiple requests simultaneously; (3) move sampling to GPU (torch.multinomial on CUDA); (4) use a C++ serving runtime (TensorRT-LLM, vLLM's C++ backend) that eliminates the Python hot path.
Napkin Math: CPU tokenizer (Python
transformers): Tokenizing 512 tokens: ~25 ms (pure Python, regex-heavy). Detokenizing 1 token: ~0.5 ms. Sampling (Pythontorch.multinomialon CPU): ~2 ms (includes GPU→CPU transfer of logits). Total CPU per token: ~27.5 ms. GPU forward pass: 7.8 ms. Pipeline: $27.5 + 7.8 = 35.3$ ms/token → 28.3 tokens/s theoretical, ~15 tokens/s with Python overhead and GIL contention. After optimization: Rust tokenizer: 0.5 ms (50× faster). GPU-side sampling: 0.1 ms (no transfer). Detokenize: 0.05 ms (Rust). Total CPU per token: 0.65 ms. Pipeline: $0.65 + 7.8 = 8.45$ ms/token → 118 tokens/s — a 7.9× improvement, now approaching the GPU's theoretical bandwidth limit.
📖 Deep Dive: Volume I: Frameworks
</details>Interviewer: "A junior engineer is fine-tuning a 7B model on a single A100 80 GB. They report: 'The model is only 14 GB in FP16, but training OOMs at batch_size=1. How can a 14 GB model not fit on an 80 GB GPU?' Walk them through where the other 66 GB went."
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "The model must be loading in FP32 — that's 28 GB" or "The batch size is too large." Even FP32 weights are only 28 GB, and they said batch_size=1. The real memory consumer is invisible in model.parameters().
Realistic Solution: The Adam optimizer stores two additional copies of every parameter: the first moment (mean of gradients, $m_t$) and the second moment (variance of gradients, $v_t$), both in FP32. Combined with the FP32 master copy of weights (needed for mixed-precision training), the optimizer state is the dominant memory consumer. The full breakdown: (1) FP16 model weights: 7B × 2 = 14 GB; (2) FP32 master weights (for optimizer update): 7B × 4 = 28 GB; (3) FP32 gradients: 7B × 4 = 28 GB; (4) FP32 Adam $m$: 7B × 4 = 28 GB; (5) FP32 Adam $v$: 7B × 4 = 28 GB. Total: 126 GB — 1.6× the GPU's memory, and we haven't even counted activations. The 14 GB model "expands" to 126 GB during training because the optimizer needs 16 bytes per parameter (FP16 weight + FP32 master + FP32 grad + FP32 m + FP32 v = 2 + 4 + 4 + 4 + 4 = 18 bytes, but the FP16 weight is a view of the FP32 master, so effectively 16 bytes of additional state). Fix: use LoRA (only optimize ~0.1% of parameters), 8-bit Adam (halves optimizer state), or gradient checkpointing + CPU offloading.
Napkin Math: Full fine-tuning memory: FP16 weights: 14 GB. FP32 master weights: 28 GB. FP32 gradients: 28 GB. Adam $m_t$ (FP32): 28 GB. Adam $v_t$ (FP32): 28 GB. Activations (batch=1, seq=2048, 32 layers): ~4 GB. CUDA workspace + fragmentation: ~2 GB. Total: ~132 GB on an 80 GB GPU → OOM. LoRA (rank=16) memory: Trainable params: $2 \times 32 \times 2 \times 4096 \times 16 = $ 8.4M (0.12% of 7B). LoRA optimizer state: 8.4M × 12 bytes = 101 MB. Frozen FP16 weights: 14 GB. Activations: 4 GB. Total: ~18.1 GB — fits on 80 GB with room for batch_size=16. The optimizer state went from 84 GB to 101 MB — a 830× reduction.
📖 Deep Dive: Volume I: Model Training
</details>Interviewer: "You're training a 175B model across 512 GPUs spanning two data center buildings connected by a 400 Gbps WAN link. During a checkpoint at step 100,000, the WAN link goes down for 8 seconds. Building A's 256 GPUs complete their checkpoint writes. Building B's 256 GPUs detect the partition and also write a checkpoint — but their gradient AllReduce was incomplete when the link dropped. Both buildings think they have a valid step-100,000 checkpoint. How do you recover without losing more than 1,000 steps of work?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "Just use Building A's checkpoint — it completed first" or "Average the two checkpoints." Building A's checkpoint may also be inconsistent (it completed the write but the AllReduce that preceded it was interrupted), and averaging inconsistent optimizer states produces garbage.
Realistic Solution: This is a distributed consensus problem applied to ML training. During the AllReduce at step 100,000, each GPU contributes its local gradients. The WAN failure means the AllReduce was partitioned: Building A's GPUs reduced among themselves (256-GPU partial reduce), and Building B's GPUs did the same. Both partitions applied a partial gradient update — each building's model diverged from the other at step 100,000. Neither checkpoint is valid for the full 512-GPU training run. Recovery requires: (1) Identify the last consistent checkpoint — step 99,000 (the previous checkpoint, completed before the partition). (2) Validate consistency — compare the model weight checksums from both buildings at step 99,000; they must be bit-identical. (3) Replay from step 99,000 — lose 1,000 steps but guarantee consistency. Prevention: (1) implement a two-phase checkpoint protocol — Phase 1: all ranks write to local storage; Phase 2: a global barrier confirms all ranks completed; only then is the checkpoint marked valid. If the barrier fails, the checkpoint is discarded. (2) Use asynchronous checkpointing that snapshots model state between AllReduce calls, when the model is in a globally consistent state.
Napkin Math: Cost of losing 1,000 steps: 512 GPUs × ~4 s/step × 1000 steps = 2,048,000 GPU-seconds = 569 GPU-hours. At $3.50/hr: $1,992 of lost compute. Cost of using an inconsistent checkpoint (if not caught): Building A's model diverges from Building B's by one full gradient step computed on half the data. The effective learning rate doubled (each building applied the full LR to a half-batch gradient). Over subsequent steps, the inconsistency compounds — within ~500 steps, the model may diverge to an unrecoverable state, wasting all compute from step 100,000 onward. If caught at step 110,000: lost = 10,000 steps = $19,920. Two-phase checkpoint overhead: Global barrier across WAN: ~50 ms (one round-trip). Checkpoint write: ~60 s for 175B model. Overhead: $50\text{ ms} / 60\text{ s} = 0.08%$ — negligible. The $0.08%$ overhead prevents a potential $20K loss.
📖 Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're running two models concurrently on one H100 80 GB using MPS (Multi-Process Service): a 7B LLM for chat and a 3B vision encoder for image understanding. Each model alone achieves its expected throughput. But when both run simultaneously, the LLM's decode throughput drops by 60% and the vision model's latency doubles. GPU compute utilization shows only 45%. What shared resource are they fighting over?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "They're competing for Tensor Cores — use MPS to partition the SMs" or "80 GB isn't enough for both models." The models total only 20 GB (14 + 6), leaving 60 GB free. And SM partitioning via MPS doesn't help because the bottleneck isn't compute.
Realistic Solution: They're saturating HBM bandwidth. LLM decode is memory-bandwidth-bound: it reads the entire 14 GB of weights to produce each token, consuming $14\text{ GB} \times \text{tokens/s}$ of bandwidth. The vision encoder's convolutional layers are also bandwidth-hungry (low arithmetic intensity for depthwise convolutions). The H100 has 3.35 TB/s of HBM bandwidth — a single shared resource that both models must share. When the LLM reads 14 GB of weights (consuming 3.35 TB/s for ~4.2 ms), the vision model's memory requests are queued. When the vision model reads its weights, the LLM's next token is delayed. They're time-multiplexing the memory bus, and neither can achieve full bandwidth. This is the memory bandwidth wall — the most common bottleneck when co-locating models on a single GPU. Fix: (1) quantize the LLM to INT4 (3.5 GB weights → 4× less bandwidth per token, leaving more for the vision model); (2) time-slice rather than co-locate: run the LLM for 10 ms, then the vision model for 10 ms, avoiding contention; (3) use separate GPUs — the bandwidth isolation is worth the extra hardware cost.
Napkin Math: H100 HBM bandwidth: 3.35 TB/s (shared). LLM alone: 14 GB weights × 128 tokens/s = 1.79 TB/s bandwidth demand (53% of peak). Achieves ~128 tokens/s. Vision model alone: 6 GB weights × 30 images/s = 180 GB/s bandwidth demand (5.4% of peak). Achieves ~30 images/s. Both concurrent: Total demand = $1.79 + 0.18 = $ 1.97 TB/s — only 59% of peak, so why the slowdown? Because memory requests are interleaved, not perfectly pipelined. Each model's access pattern (sequential weight reads) is optimized for full-bandwidth streaming. Interleaving breaks the streaming pattern, causing HBM page conflicts and row buffer misses. Effective bandwidth drops to ~2.2 TB/s (65% of peak). LLM gets $2.2 \times (1.79/1.97) = $ 2.0 TB/s → $2.0 / 14 = $ 143 tokens/s... but the interleaving adds ~40% latency overhead per access, so actual = ~51 tokens/s (60% drop from 128). Quantizing LLM to INT4: weight reads drop to 3.5 GB × 128 = 448 GB/s. Total demand: $0.448 + 0.18 = 0.63$ TB/s — only 19% of bandwidth. Both models run at full speed.
📖 Deep Dive: Volume I: Hardware Acceleration
</details>Interviewer: "Your image classification model achieves 91% accuracy in the training notebook but only 72% when deployed to a Flask serving endpoint on the same A100 GPU. Same model weights, same test images (you verified by saving the raw bytes). The training team uses PyTorch, the serving team uses ONNX Runtime. What's the most likely preprocessing bug?"
<details> <summary><b>🔍 Reveal Answer</b></summary>Common Mistake: "ONNX Runtime must have a conversion bug" or "The model needs to be re-exported." ONNX conversion for standard architectures is well-tested and unlikely to cause a 19-point drop. The model is fine — the input is wrong.
Realistic Solution: The serving pipeline applies different input normalization than training. The most common variant: the training pipeline normalizes images to [0, 1] range and then applies ImageNet mean/std normalization ($\mu = [0.485, 0.456, 0.406]$, $\sigma = [0.229, 0.224, 0.225]$). The serving pipeline either: (1) forgets the mean/std normalization entirely (passes [0, 1] values directly); (2) applies normalization but in the wrong channel order (RGB vs BGR — OpenCV loads BGR by default, PIL loads RGB); or (3) normalizes to [-1, 1] instead of using ImageNet statistics (a common TensorFlow convention applied to a PyTorch-trained model). Any of these produces inputs that are statistically different from what the model saw during training. The model still gets some predictions right (72%) because the spatial structure of the image is preserved — the model can still detect edges and shapes — but the magnitude and distribution of activations are wrong. Fix: extract the exact preprocessing pipeline from the training code (including library-specific defaults) and replicate it byte-for-byte in the serving code. Add an assertion that compares the preprocessed tensor's mean and std against expected values.
Napkin Math: ImageNet normalization: pixel value 128 (mid-gray) → $(128/255 - 0.485) / 0.229 = (0.502 - 0.485) / 0.229 = 0.074$. Without normalization: the model receives 0.502 instead of 0.074 — a 6.8× magnitude error. With BGR instead of RGB: the red channel's mean (0.485) is applied to the blue channel (which has mean 0.406). Error per pixel: $(0.406 - 0.485) / 0.229 = -0.345$ systematic bias in the blue channel. Over a 224×224 image: every one of the 50,176 pixels has a ~0.3 bias — the model sees a systematically tinted image. For a model trained on correctly normalized inputs, this is equivalent to adding a colored filter to every image. The 72% accuracy (vs 91%) means the model is robust enough to classify ~79% of "easy" images despite the bias, but fails on the ~21% that require precise color or intensity discrimination.
📖 Deep Dive: Volume I: Model Training
</details>Interviewer: "Your inference server runs 4 model replicas on 4 A100 GPUs within a single Python process using threads. Each replica should handle 100 req/s (400 req/s total). But total throughput plateaus at 110 req/s regardless of how many replicas you add. nvidia-smi shows all 4 GPUs at ~25% utilization. Adding a 5th replica doesn't increase throughput at all. What's the ceiling?"
Common Mistake: "The network or storage is the bottleneck" or "We need to increase the batch size." Network and storage are fine (the server isn't even close to saturating them), and batch size doesn't help when the GPUs are underutilized.
Realistic Solution: The Python Global Interpreter Lock (GIL) is the ceiling. CPython allows only one thread to execute Python bytecode at a time. While CUDA kernel launches release the GIL (allowing GPU work to proceed in parallel), the Python-side preprocessing (tokenization, tensor construction, result postprocessing) for each request requires holding the GIL. With 4 threads competing for the GIL, each thread gets ~25% of the CPU time for its Python work — explaining the ~25% GPU utilization (each GPU is starved of new work 75% of the time). The total throughput is bounded by how fast a single CPU core can execute the Python preprocessing pipeline: if preprocessing takes 9 ms per request, the GIL-limited throughput is $1000 / 9 \approx 111$ req/s — matching the observed 110 req/s. Adding more replicas just adds more threads competing for the same GIL. Fix: (1) use multiprocessing instead of threading — each process has its own GIL; (2) use a C++ serving runtime (Triton Inference Server, TensorRT-LLM) that bypasses the GIL entirely; (3) minimize Python-side work by moving tokenization and postprocessing to compiled extensions.
Napkin Math: Per-request Python work: tokenization (3 ms) + tensor creation (2 ms) + result decoding (1 ms) + HTTP response (1 ms) = 7 ms of GIL-holding time. GPU forward pass: 10 ms (releases GIL). Single thread: 1 request every $7 + 10 = 17$ ms → 59 req/s. GPU utilization: $10/17 = 59%$. 4 threads (GIL-limited): GIL serializes the 7 ms Python portions. Total GIL demand: $4 \times 7 = 28$ ms per 17 ms window — GIL is oversubscribed by 1.65×. Effective throughput: $1000 / 7 = $ 143 req/s theoretical GIL limit, but with GIL acquisition overhead (~2 ms per context switch × 4 threads): effective = ~110 req/s. Each GPU gets $110/4 = 27.5$ req/s × 10 ms = 275 ms of work per second → 27.5% utilization. Multiprocessing fix (4 processes): Each process has its own GIL. Per-process throughput: 59 req/s. Total: $4 \times 59 = $ 236 req/s — a 2.1× improvement. GPU utilization rises to 59% each.
📖 Deep Dive: Volume I: Frameworks
</details>memory-hierarchy architectureThe Scenario: We are deploying a 200B parameter model on an NVIDIA GH200 Grace Hopper Superchip. The spec sheet says it has 72 GB of HBM3 and 480 GB of LPDDR5X CPU RAM, sharing a unified memory space via a 900 GB/s NVLink-C2C connection. The weights are 400 GB in FP16. Your colleague says, 'Great, the weights fit entirely in the unified memory, so we can serve this on a single chip without tensor parallelism.' What is the throughput reality of this architecture?
<details> <summary><b> 🚨 Reveal the Bottleneck</b></summary>Common Mistake: "Unified memory means the GPU can read all 480 GB at HBM speeds." Unified memory means the GPU can address the CPU memory directly without explicit copies, but it does NOT bypass the physical bandwidth limitations of the interconnect.
Realistic Solution: The GH200 connects the GPU to CPU RAM via a 900 GB/s bidirectional NVLink-C2C. During autoregressive decoding, the GPU must read the entire 400 GB model to generate a single token. Since 72 GB is in HBM3 (at 4 TB/s) and the remaining 328 GB is in LPDDR5X, the GPU must fetch 328 GB over the NVLink-C2C for every single token.
The bottleneck shifts from HBM bandwidth to the C2C link bandwidth. At 900 GB/s, reading 328 GB takes a massive 364 milliseconds per token, capping your throughput at less than 3 tokens per second—completely unacceptable for production serving. To serve a 200B model efficiently, you still need tensor parallelism across multiple GPUs to keep all weights in HBM. GH200's extended memory is phenomenal for offloading KV-caches or running massive graph databases, but it is a trap for dense LLM decode weights.
Napkin Math: 400 GB model. 72 GB in HBM (4 TB/s) = 18ms. 328 GB in LPDDR5X accessed via C2C (900 GB/s) = 364ms. Total time per token = ~382ms. Max throughput = 2.6 tokens/sec. Conversely, spreading the 400 GB across 8x H100s (80GB each) gives an aggregate HBM bandwidth of 26.8 TB/s. 400 GB / 26.8 TB/s = 15ms per token (66 tokens/sec).
📖 Deep Dive: Volume I: Hardware Acceleration
</details>training storageThe Scenario: To train a 1-Trillion parameter model on 128 H100 GPUs, you use DeepSpeed ZeRO-Infinity to offload optimizer states and gradients to NVMe SSDs. You buy top-of-the-line PCIe Gen5 NVMe drives rated at 14 GB/s read/write. During the backward pass, GPU utilization drops to 12%. Where is the IO bottleneck?
<details> <summary><b> 🚨 Reveal the Bottleneck</b></summary>Common Mistake: "14 GB/s is too slow, we need more NVMe drives." While true, adding more drives won't help if you hit the root bottleneck: the PCIe switch topology and CPU root complex.
Realistic Solution: ZeRO-Infinity streams massive amounts of data from the GPU to the NVMe drives. On a standard 8-GPU node, the GPUs communicate with each other via NVSwitch, but to reach the NVMe drives, traffic must go through the PCIe switch network and often traverse the CPU root complex.
A typical dual-socket server might have 128 PCIe Gen5 lanes total from the CPUs. 8x H100s already consume 128 lanes (16 lanes each). The NVMe drives are competing for the exact same PCIe lanes and CPU interconnect bandwidth (UPI). If you try to write 8x GPU gradients to 8x NVMe drives simultaneously, the CPU's PCIe root complex becomes totally saturated. The GPUs stall waiting for DMA transfers to complete over a congested bus.
Napkin Math: 1T parameters = 2 TB of gradients (FP16). Sharded across 128 GPUs = 15.6 GB per GPU. At the end of a layer's backward pass, the GPU must write its 15.6 GB shard to NVMe. Theoretical NVMe speed: 14 GB/s -> ~1.1s. But if 4 GPUs on one CPU socket try to write 14 GB/s (56 GB/s total), they hit the CPU's internal bus limit. The effective transfer drops to ~3-4 GB/s per GPU, creating a 4-5 second stall per layer.
📖 Deep Dive: Volume II: Distributed Training
</details>parallelism memoryThe Scenario: You are using Ring Attention to train a model with a 1-million token context window across 64 GPUs. The sequence is chunked evenly (15,625 tokens per GPU). You use FlashAttention-2 locally on each GPU. Yet, profiling shows GPU 0 runs out of memory, while GPU 63 has 40 GB of free VRAM. Why does an evenly chunked sequence cause an asymmetric memory footprint?
<details> <summary><b> 🚨 Reveal the Bottleneck</b></summary>Common Mistake: "Ring Attention shouldn't have an imbalance; the communication is a perfect ring." The communication is symmetrical, but the causal mask of the attention mechanism is not.
Realistic Solution: In causal autoregressive modeling (like GPT), a token can only attend to itself and prior tokens. GPU 0 holds tokens [0 to 15,624]. These tokens can only attend to tokens within GPU 0's chunk. The upper triangle of the attention matrix is masked out. GPU 63 holds tokens [984,375 to 1,000,000]. These tokens must attend to all 1 million tokens that preceded them.
Even with FlashAttention, the backward pass must store the Log-Sum-Exp (LSE) statistics and intermediate values for the gradients. Because GPU 63 computes attention against 64x more KV blocks than GPU 0, its activation memory for the backward pass is drastically higher. GPU 63 OOMs while GPU 0 is mostly idle.
Napkin Math: For a single attention head doing the backward pass, the memory required scales with the number of unmasked query-key comparisons. GPU 0 computes ~(15K)^2 / 2 unmasked comparisons. GPU 63 computes ~15K * 1M - (15K)^2 / 2 comparisons. GPU 63 is doing nearly 64x the math and storing 64x the block-level intermediate statistics. You must use load-balancing sequence parallelism (like Striped Attention) where each GPU gets interleaved chunks.
📖 Deep Dive: Volume I: Frameworks
</details>hardware thermalThe Scenario: Your H200 cluster is processing a massive batch inference job. nvidia-smi shows the GPU core temperature is a very safe 68°C. However, your memory bandwidth benchmark shows you are only achieving 2.8 TB/s instead of the rated 4.8 TB/s. What thermal component are you failing to monitor?
Common Mistake: "The memory bandwidth must be bottlenecked by the PCIe bus." PCIe is irrelevant for internal HBM bandwidth.
Realistic Solution: You are monitoring the GPU Core temperature, but ignoring the HBM Junction Temperature. HBM3e is stacked 8 to 12 layers high directly adjacent to the GPU die. Because it is a dense 3D stack, it is incredibly difficult to cool.
HBM modules have a strict maximum operating temperature (typically around 95°C). While the GPU die might be at 68°C, a heavy memory-bound workload (like large-batch LLM prefill) generates massive heat within the memory stacks. If the HBM junction hits its thermal limit, the memory controller automatically downclocks the memory frequency to prevent physical damage, silently tanking your memory bandwidth while the core GPU temperature looks perfectly healthy.
Napkin Math: Theoretical HBM3e bandwidth = 4.8 TB/s. If the HBM stack hits 95°C, the memory controller might drop the clock from ~6 GHz to ~3.5 GHz. Effective bandwidth drops proportionally: 4.8 * (3.5/6.0) = ~2.8 TB/s. You lose 40% of your performance without a single error message.
📖 Deep Dive: Volume II: Sustainable AI
</details>compiler architectureThe Scenario: To escape the GPU shortage, you migrate a PyTorch vision model to AWS Trainium. The instance is 50% cheaper per FLOPS than an A100. However, the first epoch takes 48 hours to start, and when it does, it runs 3x slower than the A100. The code is exactly the same standard PyTorch. What went wrong?
<details> <summary><b> 🚨 Reveal the Bottleneck</b></summary>Common Mistake: "Trainium just has slower interconnects than NVLink." The interconnect isn't the issue here; the execution paradigm is.
Realistic Solution: You hit the Graph Compilation Wall. Unlike NVIDIA GPUs where PyTorch executes eagerly (kernel by kernel), AI accelerators like Trainium, Inferentia, and TPUs rely on XLA (Accelerated Linear Algebra) or Neuron compilers. They must capture the entire computation graph, optimize it, and compile it into specific machine code for their systolic arrays.
Two things happened:
if x.sum() > 0), the compiler cannot create a single static graph. It is forced to re-compile the graph on the fly during training (JIT thrashing), or fall back to the host CPU for those operations. This destroys throughput.Napkin Math: If an XLA compilation takes 30 seconds, and your data loader yields a slightly different sequence length every batch due to bucketing (e.g., 512, then 514, then 500), the Neuron compiler triggers a recompile every step. 1,000 steps * 30 seconds = 8.3 hours of pure compiling, yielding a catastrophic tokens/second rate.
📖 Deep Dive: Volume I: ML Frameworks
</details>