interviews/cloud/02_network_and_distributed.md
The domain of the AI Infrastructure Engineer. This round tests your understanding of what happens when a model exceeds the capacity of a single node: 3D parallelism, network topologies, and fault tolerance.
โ Add a Flashcard (Edit in Browser) โ see README for the template.
Interviewer: "We are training a 30B parameter model using standard Data Parallelism on 80GB GPUs. The model weights are 60GB, but the system OOMs instantly on step 1. Why?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "60 GB fits in 80 GB, so it should work. Maybe the batch size is too large." This ignores the elephant in the room.
Realistic Solution: You forgot to account for the Optimizer State. An optimizer like Adam requires 8 bytes per parameter (for the first and second moments) plus 4 bytes for a master FP32 copy of the weights. That adds 12 bytes per parameter on top of the FP16 weights. You must use ZeRO (Zero Redundancy Optimizer) or FSDP to shard these states across the workers instead of replicating them.
Napkin Math: 30B params ร 2 bytes (FP16 weights) = 60 GB. But Adam needs: 30B ร 4 bytes (FP32 master) + 30B ร 4 bytes (moment 1) + 30B ร 4 bytes (moment 2) = 360 GB. Plus 60 GB gradients. Total: 480 GB per GPU โ 6ร what you have.
Key Equation: $\text{Memory}_{Adam} = \text{Params} \times (2 + 4 + 4 + 4 + 2) = 16\ \text{bytes/param}$
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We implemented Pipeline Parallelism across 8 GPUs. However, our profiler shows the GPUs are only active 50% of the time, sitting idle while waiting for the previous GPU to finish its layer. How do we increase utilization without changing the hardware?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "We should switch to Data Parallelism." DP won't work if the model doesn't fit on one GPU โ that's why you used PP in the first place.
Realistic Solution: You need to implement microbatching. By splitting the global batch into smaller microbatches, GPU 1 can process microbatch 2 while GPU 2 processes microbatch 1. This overlaps computation and reduces the "Pipeline Bubble" fraction.
Napkin Math: Bubble fraction = $(P-1)/(M+P-1)$ where $P$ = pipeline stages, $M$ = microbatches. With $P=8$ and $M=1$: bubble = 87.5%. With $M=32$: bubble = 17.9%. With $M=64$: bubble = 9.8%. You need $M \gg P$ to keep GPUs busy.
Key Equation: $\text{Bubble Fraction} = \frac{P - 1}{M + P - 1}$
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We upgraded our CPUs to H100 GPUs, giving us a 500x speedup in raw matrix math. However, our end-to-end training throughput only increased by 20x. Where did the other 480x of our hardware investment go?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The PCIe bus is bottlenecking the data transfer." PCIe can be a factor, but the issue is more fundamental.
Realistic Solution: The Acceleration Wall (Amdahl's Law). Hardware acceleration only speeds up the parallelizable fraction ($p$) of the workload. If data loading, KV-cache updates, or Python overhead take even 5% of the step time ($p=0.95$), your maximum theoretical speedup is capped at $1/(1-0.95) = 20\times$. The serial bottlenecks will always cap the parallel gains.
Key Equation: $\text{Speedup}_{\max} = \frac{1}{(1 - p) + \frac{p}{S}}$ where $p$ = parallelizable fraction, $S$ = speedup of parallel part
๐ Deep Dive: Volume I: Hardware Acceleration
</details>Interviewer: "We have 1,024 GPUs. How do you allocate the dimensions for Data ($D$), Tensor ($T$), and Pipeline ($P$) parallelism for a 175B model?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Split evenly: $D=T=P$ or just use Data Parallelism for everything." Even splits ignore physical constraints; pure DP won't fit the model.
Realistic Solution: You solve a physical constraint satisfaction problem. $T$ is strictly bounded by the NVLink domain (usually $T=8$ per node). $P$ is bounded by the number of transformer layers and the microbatch count required to hide the bubble (e.g., $P=16$). Data parallelism ($D$) gets the remainder. Total GPUs = $D \times T \times P$, so $D = 1024 / (8 \times 16) = 8$.
Napkin Math: 175B params ร 2 bytes = 350 GB weights. One H100 has 80 GB. Minimum TP to fit weights: $350/80 \approx 5$, round up to $T=8$ (NVLink domain). With 96 transformer layers and $P=16$ stages: 6 layers per stage, bubble = $(16-1)/(M+16-1)$. Need $M \geq 45$ to keep bubble under 25%. $D = 1024/(8 \times 16) = 8$ data-parallel replicas.
Key Equation: $\text{Total GPUs} = D \times T \times P$
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We tried to scale our 70B model training by spreading Tensor Parallelism (TP) across two server racks connected by 100 Gbps Ethernet. Training speed immediately dropped to zero. What did we misunderstand about network topology?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "100 Gbps should be enough bandwidth for 8 GPUs." This confuses inter-node bandwidth with intra-node bandwidth.
Realistic Solution: You failed the "Jeff Dean Test." Tensor Parallelism requires an AllReduce operation on the activations of every single layer during the forward and backward pass. This requires the massive bandwidth of intra-node interconnects like NVLink (900 GB/s). Standard Ethernet/InfiniBand between racks (~12.5 GB/s for 100 GbE) will instantly bottleneck the GPUs. For cross-rack scaling, you must use Pipeline or Data parallelism.
Napkin Math: TP AllReduce per layer for 70B model โ 2 ร hidden_size ร batch ร bytes โ hundreds of MB per layer, 80 layers per step. NVLink: 900 GB/s โ microseconds. 100 GbE: 12.5 GB/s โ milliseconds per layer ร 80 layers = seconds of pure network wait per step.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "We placed half our GPUs in Rack A and half in Rack B. The intra-rack AllReduce is incredibly fast, but the global AllReduce crawls, even though we bought 400 Gbps InfiniBand. Where is the bottleneck?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The InfiniBand NICs must be misconfigured." Configuration is fine โ the problem is the switch topology.
Realistic Solution: Oversubscribed spine switches. A true Fat-Tree (Clos) topology guarantees non-blocking, full bisection bandwidth across the entire cluster. However, if your data center uplinks are $3:1$ oversubscribed (e.g., 3 downlinks for every 1 uplink to the spine), cross-rack traffic will instantly choke during global gradient synchronization.
Napkin Math: 8 nodes per rack, each with 400 Gbps NIC = 3.2 Tbps aggregate demand per rack. If the spine uplink is only 1.6 Tbps (2:1 oversubscription), half the gradient data is queued, doubling AllReduce time. At 3:1 oversubscription, it triples.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "For our 10B parameter model, Ring AllReduce utilizes our network perfectly. However, when we switch to a 100M parameter model, it is terribly slow despite moving far less data. Why does the 'best' algorithm fail here?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Ring AllReduce is always optimal โ something else must be wrong." Ring is bandwidth-optimal but not latency-optimal.
Realistic Solution: Ring AllReduce is bandwidth-optimal but latency-bound for small payloads. It requires $2(N-1)$ steps around the ring. For huge models, the bandwidth saturation hides the latency. For small models, the network transfer happens instantly, but the latency of hopping through $N$ nodes dominates. You must switch to a Tree reduction ($O(\log N)$ latency) for small messages.
Napkin Math: 64 nodes, 5 ฮผs per hop. Ring: $2 \times 63$ hops ร 5 ฮผs = 630 ฮผs of pure latency. Tree: $2 \times \log_2(64)$ = 12 hops ร 5 ฮผs = 60 ฮผs. For a 100M param model (200 MB), the actual data transfer at 50 GB/s takes only 4 ms โ but Ring adds 630 ฮผs of latency overhead (16% tax) vs Tree's 60 ฮผs (1.5% tax).
๐ Deep Dive: Volume II: Collective Communication
</details>Interviewer: "99 of our 100 nodes finish their backward pass in 500ms. Node 42 takes 800ms. What is the total step time for the cluster?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "About 500ms on average" or "505ms accounting for the outlier." Averaging is wrong here.
Realistic Solution: 800ms. Synchronous SGD requires a global barrier (AllReduce) to synchronize gradients before the optimizer step. The entire cluster moves at the speed of the slowest node (the straggler). You must implement robust observability to detect if Node 42 is thermal throttling, experiencing a slow PCIe lane, or if the data shard lengths are unbalanced.
Napkin Math: 100 nodes ร 500ms = 50 seconds of useful compute per step. Actual wall time = 800ms. Cluster efficiency = $500/800 = 62.5%$. That one straggler is wasting 37.5% of your entire fleet's compute budget every single step.
๐ Deep Dive: Volume II: Fleet Orchestration
</details>Interviewer: "We are scaling our training from 1,000 to 10,000 GPUs. Our current strategy pauses training for 5 minutes every hour to save a checkpoint. Is this viable at 10k scale?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "5 minutes per hour is only 8% overhead โ that's fine at any scale." This ignores how failure rates scale with node count.
Realistic Solution: No. As node count ($N$) increases, the Mean Time Between Failures ($MTBF$) decreases: $MTBF_{cluster} = MTBF_{node} / N$. At 10,000 GPUs, failures happen constantly. You must use the Young-Daly equation to balance the checkpoint overhead against the cost of lost work, which usually demands asynchronous, in-memory checkpointing to avoid stalling the training loop.
Napkin Math: If $MTBF_{node}$ = 1000 hours, then $MTBF_{cluster}$ at 1,000 GPUs = 1 hour. At 10,000 GPUs = 6 minutes. Your 1-hour checkpoint interval means you lose an average of 3 minutes of work per failure at 1k GPUs, but at 10k GPUs you fail every 6 minutes โ you'll never complete an hour of training between checkpoints.
Key Equation: $\tau_{opt} = \sqrt{2 \cdot T_{write} \cdot MTBF_{cluster}}$ (Young-Daly optimal checkpoint interval)
๐ Deep Dive: Volume II: Fault Tolerance
</details>Interviewer: "You scale your distributed training job from 64 to 256 GPUs across a standard RoCEv2 (RDMA over Converged Ethernet) network. Instead of getting a 4x speedup, your training step time actually increases. nvidia-smi shows GPU utilization dropping to 20%. What network physical phenomenon occurred?"
Common Mistake: "Assuming that because the switches have 400 Gbps bandwidth, the network scales linearly without considering packet loss and retransmission."
Realistic Solution: You hit Priority Flow Control (PFC) pause frames or severe packet loss due to incast congestion. In all-to-all communication (like AllReduce), many nodes simultaneously send data to a single switch port. Ethernet switches have shallow packet buffers. When the buffer fills, the switch drops packets (or sends PFC pause frames halting the entire upstream link). This causes severe head-of-line blocking. RDMA requires lossless networks; when packets drop, go-back-N retransmission triggers, completely stalling the GPU compute while it waits for delayed gradients.
Napkin Math: If a switch has a 32MB buffer shared across 32 ports at 400 Gbps, each port gets ~1MB. At 400 Gbps (50 GB/s), a 1MB buffer fills in
1MB / 50 GB/s = 20 microseconds. If 4 nodes blast data to 1 node simultaneously, that 20 microsecond window is easily breached, leading to immediate packet drops if congestion control isn't perfectly tuned.
๐ Deep Dive: Volume II: Network Topologies
</details>Interviewer: "Your company is building a new 512-GPU training cluster. The vendor offers two options: InfiniBand NDR (400 Gbps per port, credit-based flow control, ~1 ฮผs latency) for $2.8M, or RoCE v2 over standard Ethernet (400 GbE, PFC-based, ~2 ฮผs latency) for $1.2M. Both claim 'lossless RDMA.' Your workload is fine-tuning 70B models with FSDP across all 512 GPUs. Which do you choose, and when does the cheaper option break down?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "RoCE v2 is RDMA too, so it's equivalent to InfiniBand at half the price โ always pick RoCE." This ignores that RoCE's lossless guarantee depends on PFC (Priority Flow Control), which is fragile at scale. PFC pause frames can cascade across switches, causing head-of-line blocking that stalls unrelated flows.
Realistic Solution: For 512 GPUs doing FSDP (which uses AllGather + ReduceScatter every forward/backward pass), the critical metric is tail latency, not median latency. InfiniBand's credit-based flow control provides true lossless behavior โ the sender never transmits more than the receiver can buffer. RoCE relies on PFC, which works well within a single rack but degrades across multi-hop topologies. At 512 GPUs spanning 32+ racks, PFC storms become likely during AllReduce incast. Choose RoCE if your cluster fits in 2-3 racks with a single-tier switch; choose InfiniBand for anything larger or latency-sensitive.
Napkin Math: FSDP AllGather for 70B model: each GPU holds a 70B/512 โ 137M parameter shard โ 274 MB (FP16). AllGather collects all shards: 274 MB ร 512 = 137 GB total data moved per layer group. At 400 Gbps (50 GB/s) per link, ring AllGather takes 137 GB / 50 GB/s โ 2.74s at line rate. A 1% PFC-induced pause adds 27 ms. But tail latency matters: if the p99 RoCE latency spikes to 5ร (10 ฮผs vs 2 ฮผs per hop) due to PFC storms across 4 switch hops, the AllGather stalls accumulate to ~140 ms per step โ a 5% throughput tax that compounds over weeks of training.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You're training a 65B parameter model using pipeline parallelism with 8 stages on 8 DGX H100 nodes (64 GPUs total, 8 GPUs per node with tensor parallelism). Your team lead says 'use 16 microbatches, that's plenty.' The training run will take 3 weeks. How much GPU-time is wasted in the pipeline bubble, and how many microbatches do you actually need to keep the bubble under 5%?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "16 microbatches with 8 stages means only 8/23 โ 35% bubble โ that's acceptable for a 3-week run." Candidates often compute the bubble fraction correctly but fail to translate it into dollars and calendar time, missing that 35% bubble on 64 H100s for 3 weeks is catastrophic.
Realistic Solution: The pipeline bubble fraction is $(P-1)/(M+P-1)$ where $P$ = stages and $M$ = microbatches. With $P=8, M=16$: bubble = $7/23 = 30.4%$. To get under 5%: solve $(8-1)/(M+7) < 0.05$ โ $M > 133$. Use at least $M=140$ microbatches. However, more microbatches means smaller per-microbatch size, which can underutilize the GPU's tensor cores. You need to verify that each microbatch still has enough arithmetic intensity to saturate the H100's 989 TFLOPS (FP16).
Napkin Math: 64 H100 GPUs ร $2/GPU-hr ร 24 hr ร 21 days = $64,512 total. At 30.4% bubble: $64,512 ร 0.304 = $19,612 wasted. At 5% bubble: $64,512 ร 0.05 = $3,226 wasted โ saving $16,386. Microbatch sizing: if global batch = 4M tokens and $M=140$, each microbatch โ 28,571 tokens. With sequence length 4096, that's ~7 sequences per microbatch per pipeline stage โ enough to keep tensor cores busy.
Key Equation: $\text{Bubble Fraction} = \frac{P - 1}{M + P - 1} < 0.05 \implies M > \frac{P - 1}{0.05} - P + 1$
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "Your team wants to fine-tune LLaMA-2 7B on 8 A100 80GB GPUs in a single node. An engineer proposes standard DDP (DistributedDataParallel). Another says 'use FSDP, it saves memory.' The first engineer argues FSDP adds communication overhead for a model that already fits in memory. Who is right, and what are the exact memory numbers?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "7B parameters ร 2 bytes = 14 GB, which fits in 80 GB, so DDP is fine and FSDP is unnecessary overhead." This forgets that DDP replicates the full optimizer state on every GPU.
Realistic Solution: Both engineers are partially right. DDP replicates the entire model + optimizer on every GPU, so each A100 must hold the full memory footprint. FSDP shards parameters, gradients, and optimizer states across GPUs, dramatically reducing per-GPU memory โ but adds AllGather (before forward) and ReduceScatter (after backward) communication. For a 7B model on 8 GPUs within a single NVLink-connected node, the communication overhead is small (~900 GB/s bisection bandwidth). FSDP wins here because the memory savings let you use larger batch sizes or longer sequences, improving GPU utilization.
Napkin Math: DDP per GPU: 7B ร 2B (FP16 weights) = 14 GB + 7B ร 4B (FP32 master) = 28 GB + 7B ร 4B (momentum) = 28 GB + 7B ร 4B (variance) = 28 GB + 7B ร 2B (gradients) = 14 GB = 112 GB โ OOM on 80 GB A100! DDP actually fails here. FSDP per GPU: 112 GB / 8 GPUs = 14 GB for model state, leaving 66 GB for activations and batch data. FSDP isn't optional โ it's required.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're running synchronous data-parallel training of a 13B model across 1,024 H100 GPUs spanning 128 nodes. Each GPU has a 0.1% chance of being 'slow' on any given step (thermal throttling, ECC correction, noisy neighbor on shared NVSwitch). What fraction of steps will have at least one straggler, and how do you mitigate this without switching to async SGD?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "0.1% per GPU is tiny โ at 1024 GPUs that's maybe 1 slow GPU per step, which barely matters." Candidates underestimate how probability compounds: 0.1% per GPU becomes near-certainty at scale.
Realistic Solution: The probability that no GPU is slow on a given step is $(1 - 0.001)^{1024} = 0.999^{1024} \approx 0.358$. So 64.2% of all steps have at least one straggler. Mitigation strategies without async SGD: (1) Bounded staleness โ allow the AllReduce to proceed after 99.5% of GPUs report, using stale gradients for the remaining 0.5%. (2) Redundant computation โ run 1028 GPUs, treat 4 as hot spares, drop the slowest 4 each step. (3) Gradient compression โ reduce the communication volume so stragglers have less data to sync. (4) Profiling and eviction โ continuously monitor per-GPU step times, evict consistently slow GPUs and remap shards.
Napkin Math: $P(\text{at least one straggler}) = 1 - (1 - 0.001)^{1024} = 1 - 0.358 = 0.642$. If a straggler adds 200 ms to a 500 ms step: effective step time = 0.358 ร 500 + 0.642 ร 700 = 628.4 ms. Throughput loss: $(628.4 - 500)/500 = 25.7%$. Over a 30-day training run at $2/GPU-hr: 1024 GPUs ร $2 ร 720 hr ร 0.257 = $378,101 wasted on straggler delays.
Key Equation: $P(\text{straggler}) = 1 - (1 - p)^N$ where $p$ = per-GPU slow probability, $N$ = GPU count
๐ Deep Dive: Volume II: Fleet Orchestration
</details>Interviewer: "You're onboarding onto a distributed training team. Your tech lead mentions AllReduce, AllGather, ReduceScatter, and All-to-All in a meeting. For each of these four collectives, name one distributed training strategy that relies on it, and estimate the communication volume for 8 GPUs synchronizing a 1 GB tensor."
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "AllReduce sends the full tensor to every GPU, so the total communication is N ร tensor_size = 8 GB." This confuses the total data moved with the per-GPU send volume, and ignores that ring-based AllReduce is bandwidth-optimal.
Realistic Solution: (1) AllReduce โ Data Parallelism (DDP): sum gradients across all GPUs. (2) AllGather โ FSDP forward pass: each GPU holds a shard, AllGather reconstructs the full parameters. (3) ReduceScatter โ FSDP backward pass: reduce gradients and scatter so each GPU gets its shard's gradient. (4) All-to-All โ Expert Parallelism (MoE): route tokens to the correct expert GPU. For a ring implementation with $N$ GPUs and message size $M$: each GPU sends $M \times (N-1)/N$ in both AllReduce (as ReduceScatter + AllGather) and AllGather.
Napkin Math: 1 GB tensor, 8 GPUs, ring algorithm. AllReduce = ReduceScatter + AllGather: each GPU sends $2 \times 1\text{GB} \times 7/8 = 1.75\text{ GB}$. AllGather alone: each GPU sends $1\text{GB} \times 7/8 = 0.875\text{ GB}$ (its shard to 7 peers). ReduceScatter: each GPU sends $1\text{GB} \times 7/8 = 0.875\text{ GB}$. All-to-All: each GPU sends $(N-1)/N$ of its data = $0.875\text{ GB}$. At 900 GB/s NVLink bisection: AllReduce takes $1.75\text{GB} / 900\text{GB/s} \approx 1.9\text{ ms}$.
๐ Deep Dive: Volume II: Collective Communication
</details>Interviewer: "You have 1,024 H100 GPUs across 128 DGX nodes (8 GPUs each, NVSwitch intra-node at 900 GB/s, 400 Gbps InfiniBand inter-node). You need to train a 175B parameter GPT-style model with 96 transformer layers. Walk me through how you assign the three parallelism dimensions โ tensor ($T$), pipeline ($P$), and data ($D$) โ and justify each choice with a physical constraint."
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Maximize data parallelism for simplicity: $T=1, P=1, D=1024$." This ignores that 175B ร 2 bytes = 350 GB of weights alone won't fit on a single 80 GB GPU, so $T=1$ is physically impossible. Others suggest $T=P=D \approx 10$, ignoring that $T$ must align with the NVLink domain.
Realistic Solution: Solve inside-out from physical constraints. Tensor parallelism ($T=8$): TP requires AllReduce on activations at every layer โ this needs NVLink bandwidth (900 GB/s). The NVLink domain is exactly 8 GPUs per DGX node, so $T=8$. Going beyond 8 would cross to InfiniBand (50 GB/s), an 18ร bandwidth cliff. Pipeline parallelism ($P=8$): 96 layers / 8 stages = 12 layers per stage. Each stage holds $175B/8 \times 2B = 43.75\text{ GB}$ of weights (fits in 80 GB). Pipeline communication is point-to-point activations between adjacent stages โ low bandwidth, tolerant of InfiniBand latency. Data parallelism ($D$): $D = 1024 / (8 \times 8) = 16$. DP AllReduce happens once per step (not per layer), so it tolerates inter-node bandwidth. Verify: 16 DP replicas ร 8 TP ร 8 PP = 1024 GPUs.
Napkin Math: Memory per GPU: 175B / ($T \times P$) = 175B / 64 โ 2.73B params ร 16 bytes (Adam) = 43.7 GB โ fits in 80 GB with room for activations. Pipeline bubble: $(8-1)/(M+7)$. Need $M \geq 140$ for <5% bubble. Global batch = $M \times D \times \text{micro_batch}$ = 140 ร 16 ร 4 = 8,960 sequences โ achievable for LLM pretraining. DP AllReduce volume: 175B ร 2B / $P$ = 43.75 GB per DP group. At 400 Gbps (50 GB/s): 43.75/50 โ 0.875s โ must overlap with backward pass.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "Your colleague suggests using tensor parallelism with $T=16$ across two DGX H100 nodes to train a 70B model, arguing 'more parallelism is always better.' The model fits in 8 GPUs with $T=8$. Why is $T=16$ likely slower than $T=8$, and what is the exact communication cost per transformer layer?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Doubling tensor parallelism halves the compute per GPU, so it should be faster." This treats communication as free and ignores the NVLink domain boundary.
Realistic Solution: Megatron-style tensor parallelism splits each transformer layer's weight matrices column-wise (for the first GEMM) and row-wise (for the second GEMM), requiring two AllReduce operations per layer (one in forward, one in backward โ four total counting both passes). Within a DGX H100 node, NVSwitch provides 900 GB/s all-to-all bandwidth. Crossing to a second node drops to 400 Gbps InfiniBand (50 GB/s) โ an 18ร bandwidth reduction. The AllReduce that took microseconds intra-node now takes milliseconds inter-node, and this happens at every single layer.
Napkin Math: 70B model, hidden dim $h = 8192$, 80 layers. Per-layer AllReduce payload: $2 \times \text{batch} \times \text{seq} \times h \times 2\text{B}$ (FP16). With batch=1, seq=4096: payload = $2 \times 4096 \times 8192 \times 2 = 128\text{ MB}$ per AllReduce, 4 per layer (fwd+bwd), 80 layers = 320 AllReduces per step. $T=8$ (intra-node): 128 MB / 900 GB/s = 0.14 ms per AllReduce ร 320 = 44.8 ms total. $T=16$ (cross-node): 128 MB / 50 GB/s = 2.56 ms per AllReduce ร 320 = 819 ms total โ an 18ร communication increase that dwarfs the 2ร compute reduction.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're pretraining a 13B model on a cloud cluster. Spot instances give you between 32 and 128 GPUs at any time โ nodes can be preempted with 30 seconds notice. Your training framework uses synchronous data parallelism. How do you handle GPUs appearing and disappearing mid-training without restarting from scratch?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Just checkpoint frequently and restart with the new GPU count." This works but wastes enormous time: each restart requires loading the checkpoint, rebuilding process groups, and re-warming the data pipeline. With preemptions every ~30 minutes, you'd spend more time restarting than training.
Realistic Solution: Use an elastic training framework (e.g., TorchElastic/torchrun with --rdzv_backend=c10d). Key mechanisms: (1) Rendezvous protocol โ when a node joins or leaves, surviving workers detect the change, re-form the process group, and redistribute data shards without reloading the model. (2) Batch size adjustment โ with fewer GPUs, either reduce global batch size (changes optimization dynamics) or increase per-GPU batch size (may OOM). The safe approach: keep per-GPU batch size fixed, let global batch size float, and apply linear learning rate scaling ($\text{lr} \propto \text{global_batch}$). (3) In-memory checkpoint โ replicate model state across nodes so a preempted node's state can be recovered from peers in seconds, not minutes.
Napkin Math: 13B model, FSDP. At 128 GPUs: per-GPU shard = 13B ร 16B / 128 = 1.625 GB. At 32 GPUs: per-GPU shard = 6.5 GB โ still fits in 80 GB. Global batch at 128 GPUs: 128 ร 4 sequences = 512. At 32 GPUs: 128 sequences. LR scales: $\text{lr}{32} = \text{lr}{128} \times (128/512) = \text{lr}_{128}/4$. Rendezvous time: ~5-10 seconds. Checkpoint reload from peer memory: ~2 seconds for 6.5 GB at NVLink speed. vs. disk reload: 6.5 GB / 2 GB/s (NFS) = 3.25 seconds per GPU, but 32 GPUs hitting NFS simultaneously โ minutes.
๐ Deep Dive: Volume II: Fault Tolerance
</details>Interviewer: "Your profiler shows that for a 7B model on 64 A100 GPUs with DDP, the backward pass takes 800 ms and the AllReduce takes 600 ms, giving a step time of 1400 ms. Your colleague claims 'we should be able to overlap them and get close to 800 ms.' Under what conditions is this true, and when does the overlap break down?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "AllReduce always overlaps perfectly with backward โ just enable the flag in PyTorch DDP." Candidates assume overlap is free and complete, missing the fundamental constraint: you can only AllReduce a gradient bucket after its backward computation finishes.
Realistic Solution: PyTorch DDP partitions gradients into buckets (default 25 MB) and fires AllReduce for each bucket as soon as all gradients in that bucket are computed. Overlap works when: (1) the backward pass produces gradients steadily across time (not all at the end), and (2) the network can drain each bucket's AllReduce before the next bucket arrives. Overlap breaks down when: (a) the model has a few very large layers that produce most gradients at the end of backward (e.g., large embedding layers), (b) the network is too slow to drain buckets before they accumulate, or (c) the last bucket's AllReduce cannot overlap because there's no more computation to hide behind.
Napkin Math: 7B params ร 2 bytes = 14 GB of gradients. At 25 MB/bucket: 560 buckets. 64 GPUs on 400 Gbps IB: ring AllReduce per bucket = $2 \times 25\text{MB} \times 63/64 / 50\text{GB/s} \approx 0.98\text{ ms}$ per bucket. Total sequential AllReduce: 560 ร 0.98 ms = 549 ms โ 600 ms (matches). Backward produces buckets over 800 ms โ ~0.7 buckets/ms. Network drains at ~1/0.98 โ 1.02 buckets/ms. Since drain rate > production rate, overlap is nearly perfect โ except the last ~50 buckets (the "tail") have no computation to hide behind: tail = 50 ร 0.98 = 49 ms exposed. Achievable step time: ~800 + 49 = 849 ms, not 800 ms.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "Your company has 512 H100 GPUs in Virginia and 512 in Oregon, connected by a 100 Gbps dedicated WAN link with 60 ms RTT. Management wants to train a single 70B model across all 1,024 GPUs. Is this feasible with synchronous training, and if not, what's your architecture?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "100 Gbps is plenty of bandwidth โ just run standard DDP across both sites." This ignores that WAN latency (60 ms RTT) makes synchronous AllReduce across sites catastrophically slow, even if bandwidth is sufficient.
Realistic Solution: Synchronous AllReduce across the WAN is infeasible. A ring AllReduce with 1024 GPUs at 60 ms per cross-site hop would add seconds of latency per step. Instead, use hierarchical parallelism: run fully synchronous training within each datacenter (512 GPUs, InfiniBand, <5 ฮผs latency), then synchronize between datacenters asynchronously or with a relaxed consistency model. Options: (1) Local SGD / DiLoCo โ each site runs independent synchronous SGD for $H$ steps, then averages parameters across sites every $H$ steps. (2) Async gradient averaging โ each site pushes compressed gradient deltas over the WAN without blocking. (3) Pipeline parallelism across sites โ place different pipeline stages in each DC; only point-to-point activation transfers cross the WAN (lower bandwidth, tolerant of latency).
Napkin Math: 70B model, DDP AllReduce volume: 70B ร 2B = 140 GB. At 100 Gbps WAN (12.5 GB/s): transfer time = 140/12.5 = 11.2 seconds. Plus ring latency: if the ring crosses the WAN once, add 60 ms ร number of cross-site hops. With 1024 GPUs in a ring, ~512 hops cross the WAN: 512 ร 60 ms = 30.7 seconds of pure latency. A training step that takes 2 seconds of compute would take 33+ seconds. Local SGD alternative: sync within each DC in ~1 second, exchange compressed deltas (1% of 140 GB = 1.4 GB) every 10 steps over WAN: 1.4 GB / 12.5 GB/s = 112 ms every 10 steps = 11.2 ms amortized overhead per step.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're comparing two 8-GPU server configurations for training a 7B model with DDP. Option A: DGX H100 with NVSwitch (all-to-all 900 GB/s bisection bandwidth). Option B: a custom server with 8 H100 SXM GPUs connected via PCIe Gen5 x16 through a PCIe switch (64 GB/s per GPU, shared). What's the AllReduce time for the 7B model's gradients on each, and when is the cheaper PCIe option acceptable?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "PCIe Gen5 is 64 GB/s per direction, so 8 GPUs have 512 GB/s aggregate โ that's close enough to NVSwitch's 900 GB/s." This confuses per-link bandwidth with bisection bandwidth. A PCIe switch tree has much lower bisection bandwidth than NVSwitch's full crossbar.
Realistic Solution: NVSwitch provides a non-blocking crossbar: any GPU can talk to any other at full bandwidth simultaneously, giving 900 GB/s bisection bandwidth. A PCIe switch is a shared bus โ when multiple GPUs communicate simultaneously, they contend for switch bandwidth. The effective bisection bandwidth of a PCIe Gen5 switch with 8 GPUs is typically ~128 GB/s (2 ร 64 GB/s through the switch fabric), not 512 GB/s. For DDP AllReduce, NVSwitch is 7ร faster. PCIe is acceptable for inference (minimal inter-GPU communication) or small-model fine-tuning where compute dominates communication.
Napkin Math: 7B params ร 2 bytes (FP16 gradients) = 14 GB. Ring AllReduce: each GPU sends $2 \times 14\text{GB} \times 7/8 = 24.5\text{ GB}$. NVSwitch: 24.5 GB / 900 GB/s = 27.2 ms. PCIe switch: effective per-GPU bandwidth in ring โ 32 GB/s (half-duplex contention on shared fabric). 24.5 GB / 32 GB/s = 765 ms โ 28ร slower. If the forward+backward compute takes 500 ms, NVSwitch step time โ 527 ms, PCIe step time โ 1265 ms. PCIe throughput is 2.4ร worse.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You're training a 175B parameter model on 512 H100 GPUs using 3D parallelism. The training lead wants to checkpoint every 20 minutes (matching the cluster's estimated MTBF). Each checkpoint must be written to persistent storage. What's the checkpoint size, how long does the write take, and what's the impact on training throughput?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "175B ร 2 bytes = 350 GB, write it to NFS, done in a few seconds." This forgets the optimizer state (which is the majority of the checkpoint) and ignores that 512 GPUs writing simultaneously will saturate any shared storage system.
Realistic Solution: A full checkpoint includes FP16 weights (350 GB), FP32 master weights (700 GB), Adam moments (1,400 GB), and gradient buffers (350 GB) = 2.8 TB total. With FSDP, each GPU writes its shard: 2.8 TB / 512 = 5.47 GB per GPU. The bottleneck is aggregate storage bandwidth. A high-end parallel filesystem (Lustre/GPFS) might provide 200 GB/s aggregate write bandwidth. 2.8 TB / 200 GB/s = 14 seconds write time. But 512 GPUs issuing concurrent writes create metadata storms and I/O contention, realistically 2-3ร slower: 30-42 seconds. At 20-minute intervals, that's 2.5-3.5% overhead โ acceptable. The real solution: write to local NVMe first (each GPU has ~3.5 GB/s write), then async-flush to persistent storage.
Napkin Math: Checkpoint size: 175B ร (2 + 4 + 4 + 4) bytes = 175B ร 14B = 2.45 TB (excluding gradients). Local NVMe write: 5.47 GB / 3.5 GB/s = 1.56 seconds per GPU (parallel, no contention). Background flush to Lustre: 2.45 TB / 200 GB/s = 12.25 seconds (overlapped with training). Training step โ 2 seconds. Checkpoint every 20 min = 600 steps. Overhead with local NVMe: 1.56s / (600 ร 2s) = 0.13% โ negligible. Without local NVMe (direct to Lustre): 42s / 1200s = 3.5%.
๐ Deep Dive: Volume II: Fault Tolerance
</details>Interviewer: "You benchmark a 3B model training on A100 GPUs. Going from 1 to 8 GPUs (single node, NVLink) gives 7.6ร speedup โ nearly linear. Going from 8 to 64 GPUs (8 nodes, InfiniBand 400 Gbps) gives only 5.2ร speedup instead of 8ร. What explains the sub-linear scaling, and at what GPU count does adding more GPUs actually hurt throughput?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Sub-linear scaling means we need faster networking." While network bandwidth matters, the real issue is the communication-to-computation ratio changing as you scale โ even infinite bandwidth can't fix latency-bound collectives.
Realistic Solution: The communication-to-computation ratio determines scaling efficiency. Compute per GPU stays constant (same per-GPU batch size). But AllReduce communication grows with the number of nodes (more hops, higher latency). At 8 GPUs (intra-node), AllReduce uses NVLink at 900 GB/s with ~1 ฮผs latency โ communication is negligible. At 64 GPUs (inter-node), AllReduce crosses InfiniBand at 50 GB/s with ~5 ฮผs per hop โ communication becomes significant. The crossover point where adding GPUs hurts is when AllReduce time exceeds the compute time saved by adding one more GPU.
Napkin Math: 3B model, FP16 gradients = 6 GB. Compute per step per GPU: ~200 ms. 8 GPUs (NVLink): Ring AllReduce = $2 \times 6\text{GB} \times 7/8 / 900\text{GB/s} = 11.7\text{ ms}$. Scaling efficiency: $200/(200+11.7) = 94.5%$ โ 8 ร 0.945 = 7.56ร (matches). 64 GPUs (IB): Ring AllReduce = $2 \times 6\text{GB} \times 63/64 / 50\text{GB/s} = 236\text{ ms}$. Efficiency: $200/(200+236) = 45.9%$ โ 8 ร 0.459 = 3.67ร over 8 GPUs (not 5.2ร โ the real system uses hierarchical AllReduce to improve this). Break-even point: when AllReduce time = compute time. At ~128 GPUs, AllReduce โ 400 ms > 200 ms compute โ diminishing returns.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "Your cloud budget gets you 32 A100 80GB GPUs and 32 H100 80GB GPUs. An engineer proposes combining them into a single 64-GPU DDP training job for a 7B model. The H100 does a training step in 400 ms; the A100 takes 650 ms. What happens, and how do you actually use both GPU types productively?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "DDP averages the throughput, so we'll get something between A100 and H100 speed โ still faster than A100-only." DDP is synchronous: every GPU must complete its step before AllReduce. The entire cluster runs at A100 speed.
Realistic Solution: With naive DDP, every step takes 650 ms (the A100 speed). The 32 H100s sit idle for 250 ms per step โ wasting 38% of your most expensive hardware. Solutions: (1) Separate jobs โ run two independent training jobs, one per GPU type, and merge checkpoints periodically (Local SGD style). (2) Weighted data parallelism โ give H100s larger micro-batches proportional to their speed (H100 gets batch=6, A100 gets batch=4), so both finish at the same time. This changes the effective learning rate per GPU and requires careful tuning. (3) Heterogeneous pipeline โ assign more pipeline stages to A100s (less compute per stage) and fewer to H100s, balancing stage execution time.
Napkin Math: Naive DDP (64 GPUs, all at 650 ms): throughput = 64 samples / 650 ms = 98.5 samples/s. H100 utilization: 400/650 = 61.5%. Separate jobs: H100 throughput = 32 / 400 ms = 80 samples/s. A100 throughput = 32 / 650 ms = 49.2 samples/s. Combined: 129.2 samples/s โ 31% better than naive DDP. Weighted batching: H100 batch = 6, A100 batch = 4. H100 step โ 600 ms, A100 step โ 650 ms. Throughput = (32ร6 + 32ร4) / 650 ms = 320 / 650 ms = 492 samples/s with batch size 320 โ best option if convergence holds.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You have 256 H100 GPUs across 32 nodes, connected by a 2-tier fat-tree with 400 Gbps InfiniBand. During a 70B model AllReduce, your monitoring shows aggregate switch throughput drops from 90% link utilization to 35% as you scale from 64 to 256 GPUs. The switches aren't dropping packets. What's happening?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "If no packets are dropped, the network is healthy โ the problem must be on the GPU side." Zero packet loss doesn't mean zero congestion. Credit-based flow control (InfiniBand) prevents drops by throttling senders, which is exactly the problem.
Realistic Solution: You're experiencing congestion spreading via InfiniBand's credit-based flow control. When multiple flows converge on a single switch port (incast), the receiver's buffer fills. The switch stops issuing credits to the senders, which back-pressures upstream switches, which stop issuing credits to their senders. This "credit stall" propagates backwards through the fat-tree, throttling even flows that don't traverse the congested port (head-of-line blocking at the switch level). At 256 GPUs, the AllReduce traffic pattern creates many-to-one incast at every tier of the fat-tree simultaneously. Solutions: adaptive routing (NVIDIA SHARP, or switch-level adaptive routing), AllReduce decomposition into smaller sub-groups, or using in-network reduction (SHARP) to reduce traffic at the switch.
Napkin Math: 70B model, FP16 gradients = 140 GB. Ring AllReduce with 256 GPUs: each GPU sends $2 \times 140\text{GB} \times 255/256 = 279.5\text{ GB}$. Aggregate traffic across the network: 256 ร 279.5 GB = 71.5 TB (each byte traverses multiple links). 2-tier fat-tree with 32 spine switches, each with 400 Gbps: spine bisection = 32 ร 400 Gbps = 12.8 Tbps = 1.6 TB/s. AllReduce demands ~71.5 TB / (ideal time) flowing through the spine. At 90% efficiency (64 GPUs): AllReduce โ 2.8s. At 35% efficiency (256 GPUs): AllReduce โ 7.2s โ the 4ร GPU increase yields only 1.55ร throughput improvement.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "To avoid synchronization barriers, your team switches from synchronous to asynchronous SGD for a 13B model on 64 A100 GPUs. After 24 hours, the loss curve is 15% higher than the synchronous baseline at the same number of tokens processed. The team blames 'async is just worse.' Is that the full story, and can you quantify the staleness?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Asynchronous SGD doesn't converge as well โ switch back to synchronous." This is an oversimplification. Async SGD's problem is quantifiable: gradient staleness. The gradient a worker computes is based on parameters that are $\tau$ steps old, where $\tau$ is the staleness. The fix isn't abandoning async โ it's bounding and compensating for staleness.
Realistic Solution: In async SGD with 64 workers, when worker $i$ reads the parameter server's weights, computes a gradient, and pushes it back, other workers have updated the parameters ~63 times in between (one update per worker). Average staleness $\tau \approx N - 1 = 63$ steps. This means each gradient update is computed on parameters that are 63 steps out of date โ it's pushing the model in a direction that was correct 63 steps ago but may be wrong now. Mitigation: (1) Bounded staleness โ reject gradients older than $\tau_{max}$ (e.g., 16 steps). (2) Staleness-weighted updates โ scale the learning rate by $1/\tau$: $\theta \leftarrow \theta - (\eta/\tau) \cdot g$. (3) Local SGD โ each worker does $H$ local steps, then averages parameters globally every $H$ steps, reducing staleness to $H$ instead of $N$.
Napkin Math: 64 workers, step time = 500 ms. Synchronous: all 64 wait โ 1 update / 500 ms = 2 updates/s. Async: each worker pushes independently โ 64 updates / 500 ms = 128 updates/s. But each update has staleness $\tau \approx 63$. Effective learning rate: $\eta_{\text{eff}} = \eta / \sqrt{\tau} \approx \eta / 8$ (empirical scaling). So async processes 64ร more updates but each is ~8ร less effective โ net ~8ร more throughput, but convergence per token is worse. Local SGD with $H=8$: staleness = 8, sync every 4 seconds. Effective LR reduction: $1/\sqrt{8} \approx 0.35$, but 8ร fewer sync events. Sweet spot between sync and full async.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You split a 30B parameter GPT model across 4 A100 80GB GPUs using naive layer-wise pipeline parallelism: GPU 0 gets layers 0-23 (including the embedding), GPUs 1-2 get layers 24-47 and 48-71, GPU 3 gets layers 72-95 plus the output head. Your profiler shows GPU 0 at 78 GB memory usage while GPU 3 is at only 42 GB. What causes this imbalance, and how do you fix it?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Each GPU has 24 layers, so memory should be roughly equal โ the imbalance must be a memory leak." This ignores that activations, not parameters, dominate memory in pipeline parallelism, and the first stage must store activations for all in-flight microbatches.
Realistic Solution: In pipeline parallelism, GPU 0 (the first stage) computes its layers first and must hold its activations in memory until the backward pass reaches it โ which doesn't happen until all other stages complete their forward and backward passes. With $M$ microbatches in flight, GPU 0 stores activations for up to $M$ microbatches simultaneously. GPU 3 (the last stage) can immediately start backward after forward, so it only holds 1-2 microbatches of activations. Fix: (1) Activation checkpointing on the first stages (recompute instead of store). (2) Interleaved scheduling (1F1B) โ start backward passes earlier to free activations sooner. (3) Unequal partitioning โ give GPU 0 fewer layers to compensate for its activation memory burden.
Napkin Math: 30B model, hidden dim $h=7168$, seq_len=4096, microbatch=4, FP16. Activation per layer per microbatch: $\text{batch} \times \text{seq} \times h \times 2\text{B} \approx 4 \times 4096 \times 7168 \times 2 = 224\text{ MB}$. GPU 0 with 24 layers and $M=8$ microbatches in flight: $24 \times 8 \times 224\text{ MB} = 43\text{ GB}$ of activations + 24/96 ร 30B ร 2B = 15 GB params = 58 GB. GPU 3 with 24 layers and 2 microbatches: $24 \times 2 \times 224\text{ MB} = 10.7\text{ GB}$ + 15 GB params = 25.7 GB. With activation checkpointing on GPU 0: store only 1 activation per layer โ $24 \times 224\text{ MB} = 5.25\text{ GB}$ + 15 GB = 20.25 GB โ balanced.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're training a vision-language model on 256 A100 GPUs. Each GPU needs to load different image-text pairs at ~500 samples/second (each sample is a 256 KB JPEG + metadata). The training data lives on a shared NFS server. After scaling from 32 to 256 GPUs, data loading becomes the bottleneck โ GPUs sit idle waiting for data. What went wrong?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "NFS bandwidth is the bottleneck โ just add more NFS servers." Bandwidth may be sufficient, but NFS metadata operations (open/stat/read for millions of small files) are the real killer. NFS metadata is single-threaded on the server.
Realistic Solution: At 256 GPUs ร 500 samples/s = 128,000 IOPS of random small-file reads. NFS typically handles 10,000-50,000 metadata ops/s before the metadata server saturates. Solutions: (1) WebDataset/TFRecord โ pack thousands of samples into large sequential tar/record files. Each GPU reads a contiguous shard sequentially (1 open + sequential read vs. 500 opens/s). (2) Local SSD caching โ pre-stage data shards to each node's NVMe SSD before training. (3) Object storage with prefetching โ use S3-compatible storage with a multi-threaded prefetch pipeline that fills a local buffer 2-3 steps ahead. (4) DALI GPU-accelerated pipeline โ decode JPEGs on the GPU, overlapping decode with the previous step's compute.
Napkin Math: 256 GPUs ร 500 samples/s ร 256 KB = 32 GB/s aggregate read bandwidth. A good NFS server provides ~10 GB/s throughput โ need 3-4 NFS servers just for bandwidth. But the real bottleneck: 128,000 file opens/s. NFS metadata: ~20,000 ops/s per server โ need 6-7 servers for metadata alone. WebDataset fix: pack 1000 samples per tar shard (256 MB each). Now: 128,000 / 1000 = 128 shard reads/s, each sequential. NFS handles sequential reads at 10 GB/s easily, and metadata drops to 128 ops/s โ trivial. Local NVMe: 3.5 GB/s per node, 32 nodes. Aggregate: 112 GB/s โ 3.5ร the demand.
๐ Deep Dive: Volume II: Data Engineering
</details>Interviewer: "You're running a 70B model pretraining job on 10,000 H100 GPUs. Historical data shows each GPU has an MTBF of 20,000 hours (including NIC failures, ECC errors, thermal events). Your checkpoint-to-disk takes 45 seconds and you checkpoint every 10 minutes. Calculate the expected failures per day, the effective training throughput, and propose an architecture that achieves >90% effective utilization."
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "MTBF of 20,000 hours per GPU means we get a failure every 2 hours at 10,000 GPUs. With 10-minute checkpoints, we lose at most 10 minutes per failure โ that's manageable." This underestimates the recovery cost: it's not just lost compute, it's the time to detect the failure, restart all processes, reload the checkpoint, and re-warm the pipeline.
Realistic Solution: $\text{MTBF}_{\text{cluster}} = 20{,}000 / 10{,}000 = 2\text{ hours}$. That's 12 failures per day. Each failure requires: (1) failure detection: ~30s, (2) job kill + cleanup: ~60s, (3) node replacement/exclusion: ~120s, (4) checkpoint reload from storage: 45s (write) but read is slower with 10K GPUs hitting storage simultaneously: ~90s, (5) data pipeline re-warm: ~30s. Total recovery: ~5.5 minutes per failure. With 12 failures/day: 66 minutes of downtime + 12 ร 5 min average lost compute (half a checkpoint interval) = 60 min lost work = 126 min/day = 8.75% overhead. To hit >90% utilization: use in-memory redundant checkpointing (replicate state across 2-3 nodes, recovery in ~10 seconds), hot spare nodes (pre-loaded, ready to substitute in ~30 seconds), and elastic training (continue with N-1 GPUs while the failed node is replaced).
Napkin Math: Baseline: 24 hr ร 60 min = 1440 min/day. Checkpoint overhead: 45s every 10 min = 7.5% overhead. Failure overhead: 126 min/day = 8.75%. Effective utilization: $(1 - 0.075) \times (1 - 0.0875) = 84.3%$. Optimized (in-memory ckpt + hot spares): Checkpoint overhead: 2s every 5 min = 0.67%. Recovery time: 30s per failure ร 12 failures = 6 min/day = 0.42%. Lost compute: 2.5 min ร 12 = 30 min = 2.08%. Effective utilization: $(1 - 0.0067) \times (1 - 0.0042) \times (1 - 0.0208) = 96.8%$ โ >90% achieved.
Key Equation: $\text{Effective Utilization} = \left(1 - \frac{T_{\text{ckpt}}}{T_{\text{interval}}}\right) \times \left(1 - \frac{N_{\text{failures}} \times T_{\text{recovery}}}{T_{\text{day}}}\right) \times \left(1 - \frac{N_{\text{failures}} \times T_{\text{interval}}/2}{T_{\text{day}}}\right)$
๐ Deep Dive: Volume II: Fault Tolerance
</details>Interviewer: "You need to train a model with a 1M token context window. Standard self-attention requires $O(n^2)$ memory for the attention matrix. With hidden dim 8192 and FP16, a single attention head's score matrix for 1M tokens is 1.86 TB โ it doesn't fit on any single GPU. How do you distribute the sequence across GPUs, and what's the communication pattern?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Just use FlashAttention โ it reduces attention memory to $O(n)$." FlashAttention reduces activation memory by not materializing the full attention matrix, but the computation is still $O(n^2)$. At 1M tokens, even with FlashAttention, a single GPU would take prohibitively long to compute attention, and the KV cache for all heads still exceeds memory.
Realistic Solution: Use Ring Attention (context parallelism). Split the 1M sequence into $P$ chunks across $P$ GPUs. Each GPU holds queries for its chunk and iteratively receives key-value blocks from other GPUs in a ring pattern. In each ring step, a GPU: (1) computes attention between its local queries and the current KV block, (2) sends its KV block to the next GPU in the ring, (3) receives the next KV block from the previous GPU. After $P$ steps, every GPU has attended to all KV pairs. The key insight: computation of attention on the current KV block overlaps with communication of the next KV block.
Napkin Math: 1M tokens, $h=8192$, 128 attention heads, head_dim=64, FP16. KV per head: $1M \times 64 \times 2\text{B} = 122\text{ MB}$. All heads: $128 \times 122\text{ MB} = 15.25\text{ GB}$ for K, same for V = 30.5 GB total KV. Split across $P=32$ GPUs: 31.25K tokens per GPU. Local KV per GPU: 30.5 GB / 32 = 0.95 GB. Ring communication: each step sends 0.95 GB of KV to the next GPU. 31 ring steps ร 0.95 GB = 29.5 GB total sent per GPU. At 900 GB/s NVLink (intra-node) for 8 GPUs + 50 GB/s IB (inter-node) for 24 cross-node hops: intra-node steps: 0.95 GB / 900 GB/s = 1.06 ms. Inter-node steps: 0.95 GB / 50 GB/s = 19 ms. If attention compute per block โฅ 19 ms, communication is fully hidden. Attention FLOPs per block: $2 \times 31.25K \times 31.25K \times 64 \times 128 = 16\text{ TFLOP}$. H100 at 989 TFLOPS: 16/989 = 16.2 ms โ close but slightly less than the 19 ms transfer. Communication is almost hidden; need to tune chunk sizes or use 2 IB links.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're scaling out a new model. When you train a small 100M parameter model with Data Parallelism on 8 nodes, the epoch time is dominated by communication. However, when you switch to a very large 100B parameter model on the same 8 nodes, the communication still dominates, but the performance profile looks different. Explain why communication dominates in both cases but with different underlying reasons."
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Communication is always slow, it's just a bandwidth issue." This overlooks the fundamental difference between latency-bound and bandwidth-bound operations.
Realistic Solution: For small models (e.g., 100M parameters, FP16 gradients ~200MB), the All-Reduce operation is often latency-bound. The startup overhead (alpha) for initiating communication and traversing network hops dominates the actual data transfer time. Even with high bandwidth, the time taken to set up connections and propagate messages across nodes (due to fixed latency per hop) becomes the bottleneck.
For very large models (e.g., 100B parameters, FP16 gradients ~200GB), the All-Reduce operation becomes bandwidth-bound. The sheer volume of data being transferred over the network exhausts the available link bandwidth (beta * message_size). While latency still contributes, the time it takes to push all the bytes through the network links becomes the primary limiting factor. Optimizations like pipelining or using faster interconnects (e.g., 200/400Gbps InfiniBand/RoCE) become crucial here.
Napkin Math: Consider an All-Reduce on 8 nodes with 100Gbps (12.5 GB/s) interconnect and 100ยตs effective latency per operation.
- Small Model (200MB gradients): Time = Latency_overhead + Data_transfer_time = 100ยตs + (200MB / 12.5 GB/s) = 100ยตs + 16ms. Latency is negligible. This is incorrect. The All-Reduce communication time is roughly
alpha * log(N) + beta * MessageSize * (N-1)/N. For small messages,alpha * log(N)(latency term) dominates. Ifalphais 100ยตs, andN=8,log2(8) = 3. So latency contribution is ~300ยตs. Data transfer for 200MB on 100Gbps link is200MB / (100Gbps / 8) = 200MB / 12.5GBps = 16ms. Here, the transfer time is still larger, but for very small messages (e.g., 1MB), latency can dominate. Let's assume a practicalalphais higher, or the effective throughput is lower due to network overheads.- Revised Small Model (100MB gradients, e.g., for 50B params FP16, if sharded): All-Reduce for 8 nodes, 100ยตs per message latency, 100Gbps bandwidth. If each node sends 100MB, the total data is 800MB. The minimum time for an All-Reduce of size
SonNnodes is approximatelyalpha * log(N) + beta * S. Withalpha=100ยตsandN=8,alpha * log2(8) = 300ยตs. Withbeta=1/(100Gbps/8) = 0.08ยตs/KB, forS=100MB=100000KB,beta * S = 8ms. So total is~8.3ms. If the effectivealphadue to software stack and network hops is closer to 1-2ms, then for small messages, latency can indeed be dominant.- Large Model (200GB gradients): Time = Latency_overhead + Data_transfer_time = 300ยตs + (200GB / 12.5 GB/s) = 300ยตs + 16s. Here, bandwidth clearly dominates.
Key Equation: $T_{AllReduce} = \alpha \log_2 N + \beta \frac{N-1}{N} S$ (for ring algorithm, where $\alpha$ is latency per message, $\beta$ is inverse bandwidth, $N$ is number of nodes, $S$ is message size).
๐ Deep Dive: Volume I: Chapter 2.3.2 Collective Communication
</details>Interviewer: "You've set up a distributed training job with 16 GPUs across 4 nodes. You notice that GPU utilization is consistently low (e.g., 40-50%), even though your CPU is not maxed out, and local SSDs are very fast. The logs indicate data_loader is the bottleneck. What's the likely culprit, and how would you diagnose and mitigate it?"
Common Mistake: "It must be a CPU bottleneck in data preprocessing, or the local disk isn't fast enough." This ignores the distributed nature and potential network I/O issues.
Realistic Solution: Given that local SSDs are fast and CPU isn't maxed, the likely culprit is a network I/O bottleneck when fetching data from a shared network file system (NFS) or object storage (S3/GCS). In a distributed setting, all 16 GPUs (via their respective nodes) are concurrently trying to read data from a centralized storage system over the network. Even if individual local disk reads are fast, the aggregate network bandwidth to the shared storage, or the I/O capacity of the storage system itself, can become saturated.
Diagnosis:
nvprof/nsys or torch.profiler to identify time spent in data loading operations, particularly dataloader.next() calls.Mitigation:
num_workers in the DataLoader and use pin_memory=True to overlap data loading with GPU computation.Napkin Math: Each of 4 nodes needs to feed 4 GPUs. If each GPU requires 1GB/s of data (e.g., for a large batch size and high throughput), then each node needs 4GB/s. For 4 nodes, this is an aggregate of 16GB/s. A 10Gbps Ethernet link provides ~1.25GB/s. Even a 100Gbps link (12.5GB/s) could be a bottleneck if not configured optimally or if shared. If your shared storage is on a single 100Gbps link, it can easily be saturated by 4 nodes each pulling 4GB/s.
Key Equation: Throughput${effective}$ = min(Network${bandwidth}$, Storage${read_IOPS}$, CPU${preprocessing_speed}$)
๐ Deep Dive: Volume I: Chapter 4.2.3 Distributed Data Loading
</details>Interviewer: "Your team is implementing a very large Language Model using Pipeline Parallelism on a cluster of 64 GPUs across 8 nodes. Each node has 8 GPUs. You've observed that despite having 100Gbps interconnects, the pipeline stalls frequently, and throughput is lower than expected. You suspect the network topology is suboptimal for your communication pattern. Which common network topology might be causing this, and which would you prefer for pipeline parallelism, and why?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "All high-bandwidth networks are equally good; the problem must be in the software." This ignores the importance of matching communication patterns to network characteristics.
Realistic Solution: Pipeline parallelism involves a sequential dependency: each stage (group of GPUs) computes its part of the model and then sends its activations to the next stage. This creates a chain-like communication pattern, primarily point-to-point communication between adjacent stages in the pipeline.
A common network topology that might be suboptimal here, especially for larger clusters, is a Fat-Tree (or Clos network). While Fat-Trees offer high aggregate bandwidth and good bisection bandwidth, the path between any two arbitrary nodes might involve multiple hops through several layers of switches. For sequential point-to-point communication, this can introduce higher latency and potentially varying path lengths, leading to stalls if one link or switch becomes a bottleneck.
For pipeline parallelism, a Torus or Mesh topology would generally be preferred if available.
Why Torus/Mesh excel for Pipeline Parallelism:
Napkin Math: Consider a 64-GPU cluster (8 nodes, 8 GPUs/node). If you have an 8-stage pipeline, each stage occupies one node.
- Fat-Tree: Communication between stage 1 (Node 1) and stage 2 (Node 2) might go through 3-5 switches, incurring 300-500ns latency per hop (switch delay). Total latency could be 1-2ยตs for a single message.
- Torus (if nodes are neighbors): Communication between stage 1 and stage 2 would involve a single direct link, potentially 100-200ns latency. Even small latency differences accumulate over many micro-batches in a pipeline, significantly impacting overall throughput. For example, if each micro-batch takes 10ยตs communication time in a Fat-Tree vs 1ยตs in a Torus, and you process 10,000 micro-batches per second, that's a 90ms difference per second, which adds up.
Key Equation: Communication_Cost = $f(\text{topology}, \text{communication_pattern}, \text{message_size}, \text{latency}, \text{bandwidth})$
๐ Deep Dive: Volume I: Chapter 2.3.1 Network Topologies
</details>Interviewer: "You're managing a distributed training cluster that has grown organically. It now consists of nodes with a mix of 25Gbps and 100Gbps Ethernet interconnects, some supporting RoCE, others not. You're observing highly inconsistent training performance for Data Parallelism jobs, even for identical model architectures and batch sizes. How do you diagnose and mitigate this issue, focusing on NCCL's behavior?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "NCCL is smart, it will figure out the best path. It's probably a software bug or GPU issue." This underappreciates the complexities of heterogeneous networks and NCCL's assumptions.
Realistic Solution: NCCL (NVIDIA Collective Communications Library) is highly optimized but makes assumptions about network uniformity. In a heterogeneous environment, it can lead to suboptimal performance because:
Diagnosis:
NCCL_DEBUG=INFO or NCCL_DEBUG=WARN to see NCCL's chosen algorithms, detected bandwidths, and any warnings. NCCL_TOPO_DUMP_FILE can help visualize the detected topology.iperf3 or nccl-test (from NCCL examples) to measure point-to-point bandwidth and latency between all node pairs, specifically identifying the slowest links.Mitigation:
NCCL_IB_HCA=^mlx5_bond_0: Exclude specific slow or misbehaving interfaces.NCCL_ALGO=TREE or NCCL_ALGO=RING: Force a specific algorithm if the dynamic selection is suboptimal.NCCL_P2P_DISABLE=1: Disable P2P communication if it's causing issues (though usually not recommended).Napkin Math: A 100GB model (200GB FP16 gradients) on 16 GPUs (4 nodes, 4 GPUs/node). If All-Reduce is performed over a mix of 25Gbps and 100Gbps links, the effective bandwidth will be capped by the slowest path. If one node has a 25Gbps link (3.125 GB/s), the 200GB gradient transfer could take
200GB / 3.125GB/s = 64 secondson the bottleneck link, even if others are faster. In contrast, on a fully 100Gbps network, it would be200GB / 12.5GB/s = 16 seconds.
Key Equation: $T_{collective} \approx \frac{\text{Message Size}}{\text{Min}(\text{Bandwidth across all paths})}$
๐ Deep Dive: Volume I: Chapter 2.3.2 Collective Communication Libraries
</details>Interviewer: "You're leading the infrastructure team for training a 1-Trillion parameter model on a cluster of 2000 GPUs spread across 250 nodes. Node failures (hardware, network, OOMs) occur frequently, averaging several per day across the cluster. Restarting training from scratch after each failure is prohibitively expensive. Design a robust fault-tolerance strategy for this large-scale distributed training job."
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Just checkpoint every N steps." This simplistic approach doesn't address the coordination overhead, recovery time, or the scale of the problem.
Realistic Solution: At this scale, fault tolerance is critical and complex. A simple periodic checkpoint will likely lead to significant downtime or re-computation. A robust strategy combines several techniques:
Coordinated Checkpointing:
State Sharding for Checkpointing:
Elasticity and Resumption:
Re-computation vs. Checkpointing Trade-off:
Monitoring and Alerting:
Napkin Math:
- Model Size: 1T parameters, FP16 = 2TB weights. Adam optimizer state (FP32 master + 2 moments) = 1T * (4+8) bytes = 12TB. Total state ~14TB.
- MTBF (Mean Time Between Failures): 250 nodes, if each node has a 1% daily failure rate, then 2.5 nodes fail per day on average. This means a failure occurs roughly every $24 \text{ hours} / 2.5 \approx 9.6 \text{ hours}$.
- Checkpoint Time: If saving 14TB takes 1 hour (e.g., 400Gbps network to storage, 20 storage nodes, each providing 100GB/s), then checkpointing too frequently will kill progress.
- Optimal Checkpoint Interval: A common rule of thumb is to checkpoint such that the time spent checkpointing + time spent recomputing due to failure is minimized. If recovery takes $T_R$ and checkpoint takes $T_C$, and MTBF is $T_{MTBF}$, then optimal interval $T_{interval} \approx \sqrt{2 T_C T_{MTBF}}$. If $T_C=1 \text{ hour}$ and $T_{MTBF}=9.6 \text{ hours}$, then $T_{interval} \approx \sqrt{2 \times 1 \times 9.6} \approx \sqrt{19.2} \approx 4.4 \text{ hours}$. So, checkpointing every ~4-5 hours would be a good starting point.
Key Equation: $T_{optimal_checkpoint_interval} \approx \sqrt{2 \times T_{checkpoint} \times T_{MTBF}}$
๐ Deep Dive: Volume I: Chapter 6.3 Fault Tolerance for Distributed ML
</details>Interviewer: "You manage a large, multi-tenant GPU cluster. Users complain about unpredictable training times: sometimes jobs finish quickly, other times they take much longer, even when the cluster appears to have available GPU capacity. You suspect network congestion, but how do you verify this and implement a system-level solution to ensure more predictable network performance for critical ML workloads?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Just upgrade all network hardware." This is expensive and doesn't solve the core issue of dynamic resource contention without intelligent management.
Realistic Solution: Unpredictable performance despite available GPU capacity strongly points to network contention or inadequate Quality of Service (QoS). Different ML workloads have vastly different network demands (e.g., Data Parallelism is highly bandwidth-intensive, Model Parallelism is latency-sensitive, hyperparameter sweeps are often network-light).
Diagnosis:
torch.profiler or nsys to identify time spent in communication primitives (AllReduce, AllGather) within individual jobs. Correlate this with network metrics.iperf3 or nccl-test between nodes to measure baseline and current network performance under load.System-Level Solution (Congestion Control and QoS):
Napkin Math: A 400Gbps spine switch uplink shared by 4 racks, each with 100Gbps uplinks. If two highly network-intensive jobs (e.g., large-scale Data Parallelism) in different racks each demand 80Gbps of inter-rack traffic, they would contend for the 400Gbps spine. If a third job also demands 80Gbps, the spine link is 240Gbps/400Gbps = 60% utilized. This might seem fine, but if other management traffic or storage I/O also passes through, or if the traffic patterns are bursty, congestion can easily occur. The effective bandwidth experienced by each job could drop significantly, leading to unpredictable slowdowns.
Key Equation: $\text{Job Throughput} = \text{Min}(\text{Compute Capability}, \text{Effective Network Bandwidth})$
๐ Deep Dive: Volume I: Chapter 5.2 Resource Management for Distributed ML
</details>Interviewer: "Your company operates globally, and due to data residency regulations (e.g., GDPR, CCPA), raw training data cannot be moved outside its originating geographic region (US, EU, APAC). You need to train a single, unified, large-scale foundation model that learns from all this distributed data. Describe the architectural challenges and propose a robust solution for cross-region distributed training."
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Just copy all the data to one region and train there." This ignores data residency laws. "Train locally and average models." This is a starting point but lacks detail on WAN optimization and convergence.
Realistic Solution: Training a unified model across geographically dispersed data centers with data residency constraints presents significant challenges primarily due to high WAN latency and limited WAN bandwidth compared to internal data center networks.
Architectural Challenges:
Proposed Solution: Federated Learning with WAN Optimization & Asynchronous Aggregation
Federated Learning Architecture:
WAN Optimization for Model Updates:
Advanced Strategies:
Robustness and Security:
Napkin Math:
- Model: 100B parameters, FP16 = 200GB.
- Gradient Update: If sending full FP16 gradients, 200GB per update.
- WAN Link: 1Gbps (125MB/s) between regions.
- Latency: 200ms RTT.
- Full Gradient Transfer Time: 200GB / 125MB/s = 1600 seconds = 26.6 minutes per update (ignoring latency). With latency, this would be even worse for synchronous updates.
- Compressed Gradient (e.g., 100x compression): 2GB. Transfer time: 2GB / 125MB/s = 16 seconds. This is much more feasible for asynchronous updates, allowing updates every few minutes.
Key Equation: $T_{effective_WAN} = \frac{\text{Message Size}}{\text{WAN Bandwidth}} + \text{WAN Latency}$ (for a single transfer)
๐ Deep Dive: Volume I: Chapter 6.3.3 Distributed System Challenges and Solutions and related chapters on Federated Learning.
</details>Interviewer: "Your team is developing a novel AI accelerator chip with a custom, high-bandwidth, low-latency 3D Torus interconnect. A new distributed attention mechanism for LLMs requires an 'All-to-All-Sparse' collective operation: each of N accelerators needs to send a small, specific, non-overlapping block of data to every other accelerator, but the blocks are sparse and vary in size. Design the core communication algorithm for this 'All-to-All-Sparse' on your 3D Torus, optimizing for throughput and minimizing latency."
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Just use NCCL's All-to-All." This ignores the custom hardware and the 'sparse' aspect, which NCCL might not optimally handle.
Realistic Solution: Designing a custom collective for novel hardware requires deep understanding of the interconnect, communication patterns, and low-level primitives. The 'All-to-All-Sparse' implies that each node has unique data for every other node, similar to a standard All-to-All, but the sparse and varying size nature means we can't assume fixed-size chunks or dense packing.
Core Algorithm Design on a 3D Torus:
Understand 3D Torus Properties:
Communication Pattern Analysis (All-to-All-Sparse):
Algorithm Strategy: Multi-phase Exchange leveraging Torus Dimensions A common approach for All-to-All on a Torus is a dimension-ordered exchange. For a 3D Torus with dimensions $D_x, D_y, D_z$:
Phase 1: Exchange along X-dimension (All-to-All-X)
Phase 2: Exchange along Y-dimension (All-to-All-Y)
Phase 3: Exchange along Z-dimension (All-to-All-Z)
Optimizations for "Sparse" and "Varying Size":
Fault Tolerance (Consideration): How does the interconnect handle link failures? Does it reroute automatically? This impacts algorithm robustness.
Why this approach is good:
Napkin Math: Consider a 4x4x4 (N=64) 3D Torus. Each node needs to send a total of $63 \times \text{AvgMessageSize}$ data.
- Standard All-to-All: On a 3D Torus, the communication for an All-to-All operation can take roughly $3 \times (\text{Diameter of Dimension}) \times (\text{Message Size} / \text{Bandwidth})$. For a 4-node dimension, diameter is 2. So, roughly $3 \times 2 \times (\text{Total Data per Node} / \text{Link Bandwidth})$.
- Example: If each node sends 1KB to 63 other nodes, total 63KB data. If link bandwidth is 100GB/s, and a "hop" latency is 50ns:
- Each phase (e.g., X-dimension exchange) involves multiple steps. If it's a recursive doubling, it's
log(Dx)steps.- Total time roughly $3 \times (\text{Log of max dimension}) \times (\text{latency per hop}) + 3 \times (\text{Total Data per Node} / \text{Effective BW})$.
- For small messages, the latency term ($3 \times \log_2(4) \times 50ns = 3 \times 2 \times 50ns = 300ns$) dominates the bandwidth term. This highlights the importance of low-latency interconnects for sparse/small message All-to-All.
Key Equation: $T_{AllToAll} \approx \sum_{d=x,y,z} (\alpha_d \log N_d + \beta_d \frac{N-1}{N} S)$ (Simplified for dimension-ordered exchange on a Torus, where $N_d$ is dimension size, $\alpha_d$ is latency, $\beta_d$ is inverse bandwidth per dimension).
๐ Deep Dive: Volume I: Chapter 2.3.1 Network Topologies and advanced texts on parallel computing network algorithms.
</details>Interviewer: "You are scaling Data Parallel training from 128 to 512 GPUs. You use a standard hierarchical AllReduce. While the mathematical volume of data sent per GPU is constant regardless of cluster size, the actual network time triples when moving to 512 GPUs. Packet loss metrics show a huge spike. What physical network phenomenon is causing this?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The bandwidth is saturated." If the volume of data per GPU is constant, the bandwidth requirement per link is constant. The issue is buffers, not bandwidth.
Realistic Solution: You are experiencing TCP/RoCE Incast Congestion.
In a large distributed AllReduce (especially Tree or Halving-Doubling topologies), multiple nodes must send their gradient chunks to a single receiving node at the exact same microsecond.
When 8 leaf switches simultaneously forward data down to 1 specific destination port, the traffic bursts. The Top-of-Rack (ToR) switch has very limited physical memory buffers (e.g., 32 MB shared across all ports). During this microsecond burst, the switch buffer instantly overflows. Packets are dropped.
Because RoCEv2 (RDMA) requires a lossless network, these packet drops trigger Priority Flow Control (PFC) pause frames, which halt traffic on the upstream switches, causing a cascading "congestion spreading" effect that freezes the entire training cluster until the buffers drain.
The Fix:
Napkin Math: 8 nodes sending 1 MB chunks to 1 destination simultaneously = 8 MB burst. If the switch port only has 2 MB of dedicated egress buffer, 6 MB of packets are instantly dropped in the span of a few microseconds.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You implement Pipeline Parallelism with 8 stages. To minimize the 'bubble' (idle time), you use the 1F1B (One Forward, One Backward) scheduling strategy with 32 microbatches. It works perfectly for 100 steps. Then, suddenly, the entire pipeline stalls for 2 seconds. The computation is perfectly load-balanced. What non-compute operation broke the 1F1B rhythm?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "One of the GPUs must have thermal throttled." While true that throttling causes delays, a complete 2-second pipeline stall usually points to a synchronization barrier, not a slight clock speed reduction.
Realistic Solution: You hit the Optimizer Step Barrier.
1F1B scheduling does a beautiful job of keeping GPUs busy by interleaving the forward and backward passes of different microbatches. However, at the very end of the global batch (after all 32 microbatches have finished their backward pass), the pipeline must apply the gradients to the weights (the Optimizer Step).
The optimizer step represents a global synchronization barrier. GPU 0 cannot start the forward pass for the next global batch until GPU 7 has completely finished its backward pass for the last microbatch, and all GPUs have synchronized and updated their weights.
During this optimizer step and the subsequent pipeline refill (the new bubble), the steady-state 1F1B rhythm is broken, creating a massive, predictable stutter at the boundary of every global batch.
The Fix: Implement Interleaved Pipeline Parallelism (where each GPU is assigned multiple smaller chunks of the model to shrink the bubble further) or use asynchronous/overlapping optimizer steps if the math permits.
Napkin Math: 1F1B Bubble Fraction =
(P - 1) / M. For P=8 stages and M=32 microbatches, the bubble is7 / 32 = ~22%. That 22% of wasted time manifests entirely at the beginning and end of the global batch synchronization barrier.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You are training a 500B model using DeepSpeed ZeRO-3 across 256 GPUs. ZeRO-3 shards all parameters across all GPUs. The training speed is acceptable. However, when you increase the batch size slightly to improve utilization, the training time per step skyrockets by 500%. You haven't run out of memory (OOM). Why did a slight batch size increase destroy the network?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "Larger batches take longer to compute." Compute scales linearly. A 500% spike implies a phase transition from compute-bound to network-bound.
Realistic Solution: You caused a ZeRO-3 Prefetching / Thrashing Collapse.
ZeRO-3 works by dynamically fetching the required weight shards from other GPUs over the network just before computing a layer, and immediately discarding them after the layer finishes to save memory.
To hide this network latency, DeepSpeed uses aggressive prefetching: it starts fetching the weights for Layer N+1 while the GPU is still computing Layer N.
When you increased the batch size, the activation memory (which scales linearly with batch size) grew. To fit these larger activations, the ZeRO memory allocator had less headroom to hold pre-fetched weights. The prefetch buffer size was forced to shrink.
Because the prefetch buffer was too small, the GPU finished computing Layer N, but the weights for Layer N+1 hadn't arrived yet. The GPU stalled, waiting for the InfiniBand network. This happened for every single layer, turning a compute-bound training job into a pure network-IO-bound job.
The Fix: You must balance the memory. If you increase batch size (activations), you must either use Activation Checkpointing to free up RAM, or increase the size of the cluster so the per-GPU parameter shards are smaller, leaving enough memory for the critical network prefetch buffers.
Napkin Math: 500B model = 1TB FP16 weights. Sharded over 256 GPUs = 4 GB of weights stored per GPU. However, to compute Layer N, the GPU needs the full 15 GB weight matrix for that specific layer. If the network bandwidth is 50 GB/s, fetching the layer takes 300ms. If compute takes 200ms, and you don't have RAM to prefetch, the GPU spends 100ms completely idle per layer.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You are running Data Parallel training on a single 8-GPU DGX node. GPU 0 crashes due to a faulty memory bank. You rewrite the host script to exclude GPU 0 and launch the job on GPUs 1 through 7. The job starts, but the NCCL AllReduce immediately hangs indefinitely. Why does a 7-GPU topology fail on a DGX?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The batch size must be divisible by the number of GPUs." While math divisibility is a common issue, it would throw a shape mismatch error, not hang the network.
Realistic Solution: You broke the Physical NVLink Ring/Mesh Topology.
In an NVIDIA DGX server, the GPUs are physically connected to each other via NVSwitch and hardwired NVLink traces on the motherboard. NCCL (NVIDIA Collective Communication Library) profiles this physical hardware at startup to create an optimal logical ring or tree for passing gradients.
When you exclude GPU 0, you leave a physical "hole" in the hardware topology. NCCL attempts to build a continuous high-speed ring across GPUs 1-7. If the physical wiring relies on GPU 0 to bridge certain NVSwitch domains (depending on the specific baseboard architecture), NCCL cannot form a closed loop using only NVLinks.
Instead of falling back to the much slower PCIe bus automatically, NCCL will often hang during the ring negotiation phase, or it will construct a broken ring that deadlocks waiting for a signal from a missing hardware path.
The Fix: You must explicitly set NCCL_P2P_DISABLE=1 (forcing it to use PCIe, which destroys performance) or, more practically, you cannot use an asymmetric subset of GPUs on a tightly coupled baseboard. You must repair the node.
Napkin Math: A standard AllReduce on 8 GPUs via NVLink (900 GB/s) takes ~20ms. If forced to fallback to PCIe Gen4 routing through the CPU root complex to bypass a dead GPU, the bandwidth drops to ~32 GB/s, taking ~560ms. A 28x slowdown, effectively ruining the node.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You are using PyTorch DistributedDataParallel (DDP) across 4 GPUs. GPU 3 is slightly slower than the others due to thermal throttling. You notice that GPU 0, 1, and 2 are sitting idle for 200ms at the end of every backward pass. You know DDP overlaps communication with computation. Why isn't the overlap hiding the delay of GPU 3?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "DDP waits until the entire backward pass is finished before communicating." This is exactly what DDP doesn't do. DDP uses gradient bucketing.
Realistic Solution: You are experiencing the Global Synchronization of the Final Bucket.
PyTorch DDP works by grouping gradients into "buckets" (e.g., 25 MB each). As soon as the backward pass finishes calculating the gradients for Bucket 1, it immediately fires off an asynchronous network AllReduce for Bucket 1, while the GPU continues calculating the backward pass for Bucket 2. This perfectly overlaps compute and network.
However, for the very last bucket (which contains the gradients for the first layers of the model), there is no more compute left to overlap. Furthermore, the optimizer step cannot begin until all buckets have finished their AllReduce across all GPUs.
If GPU 3 is 200ms slower at finishing its math for the final bucket, GPUs 0, 1, and 2 will hit the hard synchronization barrier and sit completely idle. The network cannot fix a compute straggler.
The Fix: You must address the thermal throttling on GPU 3, or use an asynchronous training method (which hurts convergence) to break the strict synchronization barrier.
Napkin Math: If 3 GPUs finish in 800ms, and 1 GPU finishes in 1000ms. The effective step time is 1000ms. You are wasting 200ms * 3 GPUs = 600ms of GPU compute time every single step. At $3.00 an hour per GPU, you are throwing away thousands of dollars just waiting for the slow chip.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You're running standard Data Parallel (DDP) training on 256 GPUs. One specific node has a slightly degraded cooling fan, causing its 8 GPUs to thermal throttle and run 15% slower than the rest of the cluster. The network is perfectly healthy. By exactly how much does this single degraded node slow down the entire 256-GPU training job?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "It slows down the job by (15% / 256) since it's only one node." This assumes asynchronous execution.
Realistic Solution: It slows down the entire job by exactly 15%.
Data Parallelism uses synchronous SGD. At the end of every single backward pass, all 256 GPUs must participate in a global AllReduce operation to sum their gradients before the optimizer step can occur.
This creates a strict global barrier. The 248 healthy GPUs will finish their math at 100% speed, and then sit completely idle (0% utilization), waiting for the throttled node to finish its math and join the collective communication ring. The throughput of a synchronously parallel system is strictly dictated by the throughput of its absolute slowest component.
The Fix: You must proactively monitor hardware health (e.g., tracking AllReduce wait times per rank) and evict/replace the straggler node, or switch to an asynchronous or gradient-staleness tolerant architecture (which is mathematically riskier).
Napkin Math: 256 GPUs * $3/hr = $768/hour. A 15% slowdown wastes $115 every hour in idle compute time across the healthy nodes, vastly exceeding the cost of just replacing the $15 broken cooling fan.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You are trying to fit a 30B parameter model on a single 8-GPU node (80GB A100s) for fine-tuning. The weights in FP16 take 60GB. You enable DeepSpeed ZeRO Stage 1, which partitions the optimizer states across the 8 GPUs. However, the system still instantly OOMs on the first forward pass, even with a batch size of 1. Why didn't ZeRO-1 save you enough memory?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "ZeRO-1 reduces the memory footprint by 8x, so it should fit." ZeRO-1 only partitions the optimizer states, not the gradients or the weights.
Realistic Solution: You ran out of memory because you didn't shard the Gradients or the Model Weights.
Let's calculate the memory per GPU with ZeRO-1:
Total memory required per GPU: 60 GB + 60 GB + 45 GB = 165 GB. Your 80 GB A100 OOMs immediately.
The Fix: To fit this model on 80GB cards, you must use ZeRO Stage 3, which partitions the weights, gradients, AND optimizer states across all 8 GPUs. (Total per GPU: (60+60+360)/8 = ~60 GB, which fits nicely).
Napkin Math: ZeRO-1 memory:
(W + G) + (O / N). ZeRO-3 memory:(W + G + O) / N. For large models, W and G alone will exceed VRAM, forcing the use of Stage 3 or tensor parallelism.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You are serving a 70B model. It fits perfectly on 2x A100 (80GB) GPUs using Tensor Parallelism (TP=2). To handle more traffic, you buy 2 more A100s. Your colleague says, 'Let's just increase to TP=4, that will double our throughput because the math is split 4 ways instead of 2.' You argue that TP=4 will actually be slower than running two separate TP=2 replicas. Why?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "More GPUs always means faster math." Math scales, but communication overhead does not scale linearly in Tensor Parallelism.
Realistic Solution: You are fighting the Tensor Parallelism AllReduce Tax.
In Tensor Parallelism (Megatron-LM style), every single Transformer layer requires two AllReduce operations across all participating GPUs (one after the self-attention block, one after the MLP block).
An AllReduce operation's latency is heavily dependent on the number of devices participating in the ring/tree. By expanding from TP=2 to TP=4, you have doubled the number of synchronization barriers the GPUs must hit for every single token generated.
Because LLM decoding is already memory-bandwidth bound (not compute bound), splitting the matrix math 4 ways yields almost zero compute benefit, but incurs massive network synchronization penalties.
The Fix: You should use Data Parallelism (or Replica Parallelism) for serving. Run Replica A on GPUs 0,1 (TP=2) and Replica B on GPUs 2,3 (TP=2). This gives you exactly 2x the throughput with zero additional network overhead.
Napkin Math: A 70B model with 80 layers requires 160 AllReduces per token. At TP=2, an NVLink AllReduce might take 5ยตs (160 * 5 = 0.8ms total). At TP=4, it might take 8ยตs (160 * 8 = 1.28ms total). The network overhead increases latency by 50% without speeding up the memory-bound math at all.
๐ Deep Dive: Volume II: Distributed Training
</details>Interviewer: "You are building a 4,000 GPU cluster. You use a standard Fat-Tree InfiniBand topology. The cluster is divided into 4 pods of 1,000 GPUs each. Jobs running entirely within Pod A achieve 98% network scaling efficiency. However, when you launch a 2,000 GPU job spanning Pod A and Pod B, the network efficiency plummets to 40%. The cables are all 400 Gbps. What architectural constraint in the Fat-Tree is causing the bottleneck?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The distance between the pods adds latency." Fiber optic latency over 100 meters is microseconds. It doesn't drop efficiency by 60%.
Realistic Solution: You designed an Oversubscribed Core Network (Blocking Topology).
In a Fat-Tree topology, Leaf switches connect to Spine switches, which connect to Core switches. To save massive amounts of money on expensive Core switches and optics, network architects often use an "oversubscription ratio" (e.g., 3:1) at the higher layers.
This means that while every GPU has a dedicated 400 Gbps link to its local Leaf switch (non-blocking within the Pod), the total bandwidth connecting Pod A to Pod B is intentionally bottlenecked. If 1,000 GPUs in Pod A try to simultaneously talk to 1,000 GPUs in Pod B, the uplink cables to the Core switches can only handle 1/3rd of the traffic.
The network physically drops packets or uses hardware flow control (PFC) to pause the GPUs, forcing them to wait for bandwidth to clear.
The Fix: For massive distributed training (like LLMs), you must design a strictly Non-Blocking (1:1 Oversubscription) Clos Network, ensuring the total bisection bandwidth at the Core is equal to the sum of the edge bandwidth, regardless of the cost.
Napkin Math: 1,000 GPUs * 400 Gbps = 400 Tbps of traffic trying to leave the Pod. At a 3:1 oversubscription ratio, you only have 133 Tbps of uplink capacity. 267 Tbps of data is physically blocked at the Spine switch every second.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You have a cluster of 8 DGX nodes. Each node has 8 GPUs connected via NVLink, and 8 dedicated Mellanox InfiniBand NICs. You launch a PyTorch distributed training job. The job runs, but cross-node communication is bizarrely slow. You check nvidia-smi topo -m and see all hardware is physically connected correctly. You check the NCCL logs and see NCCL INFO: Using PCIe for cross-node communication. Why is NCCL ignoring your expensive InfiniBand network?"
Common Mistake: "The InfiniBand cables are broken." If they were broken, it would fail to route entirely. PCIe means it's routing through the host CPU.
Realistic Solution: You failed to configure the NCCL Network Interface Bindings (NCCL_SOCKET_IFNAME).
By default, NCCL probes the OS for available network interfaces. If your cluster has standard 1 Gbps Ethernet management interfaces (e.g., eth0) alongside the high-speed InfiniBand interfaces (e.g., ib0, ib1), NCCL can sometimes guess incorrectly and bind the collective communication rings to the slow Ethernet management network.
Because the Ethernet network is incredibly slow, NCCL detects that routing traffic through the CPU's PCIe bus to the Ethernet NIC is the only path available. It completely ignores the ib interfaces.
The Fix: You must explicitly tell NCCL which physical network interfaces to use by setting environment variables in your launch script: export NCCL_IB_DISABLE=0 and export NCCL_SOCKET_IFNAME=ib0,ib1... to strictly bind the communication to the high-speed hardware.
Napkin Math: InfiniBand NDR = 400 Gbps (50 GB/s). Standard Ethernet Management = 1 Gbps (125 MB/s). An incorrect environment variable silently downgraded your trillion-parameter training network speed by a factor of 400x.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You are training a model on a massive cloud VM with 8x GPUs. You download your 1 TB dataset from S3 to the VM's attached local NVMe drive (ephemeral storage). Training speeds are phenomenal. However, 12 hours into the 48-hour training run, the VM is preempted (Spot Instance). When the orchestration system brings the VM back up and resumes from the checkpoint, training crashes because the dataset file is missing. Why?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The orchestration system didn't mount the drive." It's mounted, but it's empty.
Realistic Solution: You forgot that Ephemeral Storage is physically tied to the Host hardware.
When you provision a cloud VM with "local NVMe" (like AWS instance store), that SSD is physically bolted into the server rack hosting your VM. It offers incredible bandwidth (e.g., 10 GB/s) because it bypasses the network.
However, when a Spot Instance is preempted, the cloud provider tears down your VM and gives the hardware to someone else. When your job is rescheduled 5 minutes later, it is assigned to a completely different physical server in the datacenter. The new server's local NVMe drive is completely blank. Your 1 TB dataset was destroyed when the previous VM died.
The Fix: For preemptible/Spot instances, you cannot rely on ephemeral local storage for persistent state.
Napkin Math: Re-downloading 1 TB at 10 Gbps (1.25 GB/s) takes roughly 15 minutes. Every time you get preempted, you lose 15 minutes of compute time just waiting for the dataset to arrive on the new hardware.
๐ Deep Dive: Volume II: Fault Tolerance
</details>Interviewer: "You are transferring a 2 TB model checkpoint from an AWS datacenter in Virginia to a GCP datacenter in Tokyo. You have a dedicated 10 Gbps direct fiber link. However, your scp or rsync transfer maxes out at a pathetic 250 Mbps. The link is not shared, and there is 0% packet loss. Why is standard TCP catastrophically failing to use the bandwidth over a long distance?"
Common Mistake: "AWS is throttling the egress." While egress costs money, dedicated lines aren't artificially throttled to 2%. The issue is the protocol physics.
Realistic Solution: You are a victim of the Bandwidth-Delay Product (BDP) and TCP Receive Windows.
Standard TCP requires the sender to receive an Acknowledgment (ACK) from the receiver before it can send more data. The amount of un-acknowledged data allowed in flight is dictated by the TCP Window Size.
Over a long distance (Virginia to Tokyo), the Round Trip Time (RTT) is roughly 150ms. If your OS has a default maximum TCP window size of 4 Megabytes, the sender blasts 4 MB into the fiber optic cable, and then must completely stop sending and wait 150ms for the ACK to return.
You are physically only allowed to send 4 MB every 150ms.
4 MB / 0.150 seconds = 26.6 MB/s (roughly 212 Mbps).
It does not matter if your pipe is 10 Gbps or 100 Gbps; the strict mathematical relationship between the TCP window size and the speed of light (latency) creates a hard throughput ceiling.
The Fix:
sysctl -w net.core.rmem_max=...) to massively increase the window size (BDP tuning).Napkin Math: To fill a 10 Gbps (1.25 GB/s) pipe with a 150ms latency, your TCP Window Size must be:
1.25 GB/s * 0.150s = 187.5 Megabytes. The Linux default is usually 2 MB to 6 MB. You were starving the pipe by a factor of 40x.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You are training a model across 64 GPUs on AWS using standard 100 Gbps Ethernet (TCP/IP). You notice the GPUs are only running at 60% utilization. You look at the host CPU, and all 48 CPU cores are pinned at 100% utilization during the AllReduce phase. The CPU is not doing any data loading. What is the CPU doing that is bottlenecking a 100 Gbps network?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The CPU is doing the AllReduce addition math." The GPUs do the addition. The CPU is struggling with the data transport.
Realistic Solution: You are bottlenecked by the OS Kernel TCP/IP Stack Processing.
When you use standard TCP/IP sockets over Ethernet, every single packet (typically 1500 bytes) must be processed by the Linux kernel. The CPU must calculate checksums, handle sliding windows, acknowledge packets, and physically copy the data from the network card's ring buffer into kernel space, and then copy it again into user-space memory (the GPU's memory buffer).
Pushing 100 Gigabits per second (12.5 GB/s) through the Linux kernel requires millions of interrupts and memory copies per second. The CPU cores are completely overwhelmed just executing the TCP protocol state machine, meaning they cannot feed data to the network card fast enough to saturate the 100 Gbps link. The network stalls, which makes the GPUs stall.
The Fix: You must bypass the OS kernel entirely using RDMA (Remote Direct Memory Access) via RoCEv2 (RDMA over Converged Ethernet) or EFA (Elastic Fabric Adapter on AWS). RDMA allows the network card to read/write directly to the GPU's memory without the CPU or the Linux kernel ever touching the packets, dropping CPU utilization to near 0%.
Napkin Math: 100 Gbps = 12.5 GB/s. With an MTU of 1500 bytes, the CPU must process 8.3 million packets per second. At 1 microsecond of kernel overhead per packet, that requires 8.3 dedicated CPU cores running at 100% just to run the network, completely destroying the node's balance.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You are buying a custom 8-GPU server for training. Vendor A offers 8x RTX 4090s connected entirely via PCIe Gen4 switches. Vendor B offers 8x H100s connected via NVLink and NVSwitch. Vendor A's server is much cheaper, and they claim 'PCIe Gen4 x16 gives 64 GB/s, which is plenty for training.' Why will Vendor A's server fail catastrophically at 8-GPU Data Parallel training compared to Vendor B?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "PCIe is slower than NVLink." It is slower, but the real issue is the topology, not just the raw speed.
Realistic Solution: You hit the PCIe Root Complex Bottleneck (Tree vs Mesh topology).
NVLink + NVSwitch provides a fully non-blocking mesh. GPU 1 can talk to GPU 8 at 900 GB/s at the exact same time GPU 2 talks to GPU 7.
PCIe is a tree topology. All 8 GPUs must eventually route their traffic through PCIe switches, which often bottleneck at the CPU's Root Complex. If GPU 1 tries to send 64 GB/s to GPU 2, and GPU 3 tries to send 64 GB/s to GPU 4, they both hit the same upstream PCIe switch. The bandwidth is divided.
During an AllReduce operation, all 8 GPUs are transmitting simultaneously. The PCIe bus instantly saturates, and the effective bandwidth per GPU drops from 64 GB/s to a fraction of that, causing massive network stalls during every optimizer step.
The Fix: Do not use PCIe for massively parallel multi-GPU training within a single node. You must use hardware with dedicated GPU-to-GPU interconnects (NVLink) that bypass the CPU and PCIe tree entirely.
Napkin Math: NVLink Bisection Bandwidth on an 8-GPU node = ~3,600 GB/s. PCIe Gen4 CPU Root Complex limit = ~128 GB/s (total, shared by all lanes). The NVLink topology is roughly 28x faster during a global AllReduce.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You are running a 1,000 GPU training job over a massive InfiniBand network. To maximize bandwidth, you enable Adaptive Routing (AR), allowing the network to dynamically send packets down the least congested paths. Throughput improves, but occasionally, NCCL crashes with an unrecoverable timeout error during a global AllReduce. The network cables and switches are perfectly healthy. What did Adaptive Routing do to the packets to break NCCL?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The packets got dropped by a congested switch." InfiniBand is lossless (credit-based flow control); it rarely drops packets. The problem is the order.
Realistic Solution: You caused Out-of-Order Packet Delivery that NCCL cannot handle.
Standard Ethernet/TCP handles out-of-order packets via the kernel stack, reordering them before the application sees them. InfiniBand RDMA bypasses the CPU and writes directly to GPU memory to achieve ultra-low latency. Many RDMA operations (like memory writes) strictly assume packets will arrive in the exact order they were sent.
When you enable Adaptive Routing, Packet 1 might take Path A, and Packet 2 might take a slightly faster Path B. Packet 2 arrives at the destination GPU before Packet 1. If NCCL is heavily optimized and expects contiguous memory updates to signal completion, an out-of-order arrival breaks the internal state machine of the collective communication algorithm. The GPU waits for Packet 1, stalls, and eventually triggers a hard timeout, crashing the entire training job.
The Fix: Adaptive Routing on InfiniBand requires hardware features like SHIELD (Self-Healing Interconnect Enhancement for Intelligent Datacenters) or specific NCCL ordering configurations that allow the local NIC to reorder RDMA packets in hardware before they hit the GPU memory. Otherwise, you must disable AR and rely on static ECMP routing.
Napkin Math: If Path A has 2 microseconds of congestion delay, Packet 2 easily beats Packet 1 to the destination. In a 1,000 GPU AllReduce, the statistical probability of a micro-congestion event reordering a critical packet approaches 100% over a 12-hour training run.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You are using RoCEv2 (RDMA over Ethernet) for a 64-GPU cluster. The network links are 100 Gbps. During the AllToAll phase of a Mixture of Experts (MoE) training step, the network throughput plummets, and PFC (Priority Flow Control) pause frames flood the network. The total data volume sent by any GPU is well under the 100 Gbps limit. Why is the network freezing?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The bandwidth is saturated." The prompt explicitly says the data volume is under the limit. The issue is time, not volume.
Realistic Solution: You triggered a Top-of-Rack (ToR) Switch Microburst.
In an AllToAll operation, every GPU simultaneously sends a small chunk of data to every other GPU. If 63 GPUs all try to send a 1 MB chunk of data to GPU #1 at the exact same microsecond, 63 MB of traffic instantly hits the ToR switch port connected to GPU #1.
While the average bandwidth over a second is low, the instantaneous burst is massive. A typical Ethernet ToR switch only has a few megabytes (e.g., 16 MB to 32 MB) of packet buffer memory shared across all ports. The 63 MB microburst instantly overflows the switch's buffer. Because RoCEv2 requires a lossless network, the switch fires PFC Pause Frames back to the sender GPUs, forcefully halting all transmissions and freezing the cluster until the buffer drains.
The Fix:
Napkin Math: 63 GPUs * 1 MB = 63 MB burst. ToR Switch buffer = 32 MB. The buffer overflows in less than 1 millisecond, triggering a cascading network-wide pause.
๐ Deep Dive: Volume II: Network Fabrics
</details>Interviewer: "You rent a bare-metal server with 4x GPUs. They do not have NVLink; they communicate over the motherboard's PCIe bus. You run an NCCL test, and the GPU-to-GPU bandwidth is only 12 GB/s, instead of the expected 24 GB/s for PCIe Gen4 x16. You notice the host CPU utilization is spiking during the transfer. Why is the CPU getting involved in a direct GPU-to-GPU transfer?"
<details> <summary><b>๐ Reveal Answer</b></summary>Common Mistake: "The PCIe bus is too slow." PCIe Gen4 x16 supports ~25 GB/s. The issue is the routing of the traffic.
Realistic Solution: You are blocked by PCIe Access Control Services (ACS) preventing P2P DMA.
In a multi-GPU system, GPUs use Peer-to-Peer (P2P) DMA to read and write directly to each other's VRAM over the PCIe switches, completely bypassing the CPU.
However, in many enterprise servers (especially those designed for virtualization or security), the BIOS enables PCIe ACS (Access Control Services) or IOMMU isolation. This security feature strictly forbids direct communication between PCIe devices to prevent rogue devices from attacking each other.
Because the hardware blocks the direct P2P path, NCCL is forced to fall back to routing the traffic through the host CPU's memory. GPU 1 writes to system RAM, the CPU manages it, and GPU 2 reads from system RAM. This double-copy through the CPU's memory controller halves the effective bandwidth and spikes CPU usage.
The Fix: You must enter the server's BIOS settings and explicitly disable PCIe ACS (or configure IOMMU pass-through policies) to allow the PCIe switch to route packets directly between the GPU endpoints.
Napkin Math: Direct P2P = 1 trip over PCIe (24 GB/s). CPU routing = GPU to RAM (Trip 1), RAM to GPU (Trip 2). The data crosses the bus twice, mathematically halving the maximum throughput to 12 GB/s.
๐ Deep Dive: Volume II: Infrastructure
</details>