interviews/cloud/04_data_and_mlops.md
The domain of the ML Leadership and Responsible Engineer. This round tests your ability to maintain system health over time: managing data drift, technical debt, and the Total Cost of Ownership (TCO).
β Add a Flashcard (Edit in Browser) β see README for the template.
Interviewer: "Your team rents 64 A100 GPUs to train a large vision model. After a month, the cloud bill arrives: $800,000. You pull the utilization logs and discover average GPU compute utilization was 23%. The team swears the training loop is optimized. Where did 77% of your GPU-hours go?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "The model isn't big enough to saturate the GPUs" or "We need to increase the batch size." Both assume the bottleneck is inside the training loop β it's not.
Realistic Solution: The GPU is idle because the surrounding infrastructure can't feed it fast enough. In production ML systems, the model code is roughly 5% of the system (per Google's "Hidden Technical Debt" paper). The other 95% β data ingestion, preprocessing, feature extraction, checkpointing, logging, and gradient synchronization β creates the actual bottleneck. Common culprits: CPU-bound image decoding (8 CPU cores can't decode fast enough for 8 GPUs), slow NFS reads during shuffling, synchronous checkpointing that stalls all GPUs every N steps, and Python GIL contention in the data loader.
Napkin Math: 8 A100s at 312 TFLOPS FP16 each demand ~40 GB/s of decoded training data. If your data pipeline delivers 10 GB/s (limited by CPU decoding + storage I/O), GPUs are starved 75% of the time. At $2/GPU-hour, 64 GPUs Γ 720 hours Γ 77% idle = $71,000/month wasted on idle silicon. The fix (DALI GPU decoding + NVMe staging) costs $5,000/month.
π Deep Dive: ML Operations
</details>Interviewer: "Our DevOps dashboard shows 99.99% uptime, 50ms latency, and 95% GPU utilization on our recommendation cluster. The HTTP error rate is zero. But the business team is furious because our recommendations are completely wrong. Why does the high GPU utilization mask the failure, and how can the hardware be perfectly healthy while the ML output is perfectly wrong?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "There must be a bug in the model code" or "The A/B test is misconfigured." Both assume a software failure β this is a data failure masked by hardware metrics.
Realistic Solution: The Operational Mismatch. Traditional software fails loudly (crashes). ML systems fail silently. A model experiencing Data Drift (e.g., user behavior changed due to a holiday) will continue serving predictions with full confidence. The GPU doesn't know the data is stale; it just sees matrices to multiply. A 95% GPU utilization metric just means the hardware is efficiently doing the wrong math. You must engineer statistical telemetry (e.g., KL Divergence, PSI) to monitor input distributions, because hardware metrics (utilization, latency, throughput) are completely decoupled from ML correctness.
Napkin Math: A recommendation model trained on summer shopping data sees winter holiday traffic. Input feature distributions shift by 2-3 standard deviations. The GPU continues executing 100 TFLOPS of embedding lookups and MLP layers in exactly 50ms per batch. The model's confidence scores remain 0.95+ (it doesn't know what it doesn't know), but click-through rate drops from 12% to 2%. Traditional monitoring sees: 95% GPU util, green across the board. Business sees: 83% revenue drop.
π Deep Dive: ML Operations
</details>Interviewer: "Our model achieves 95% accuracy when evaluated offline on an A100 cluster, but drops to 70% when deployed to production on T4 GPUs. The model weights and the Python feature code are identical. How do the different hardware paths cause this numerical divergence?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "The test set isn't representative of production data." Distribution shift is possible, but hardware-induced numerical divergence is a silent killer when weights and code are identical.
Realistic Solution: Hardware-induced Training-Serving Skew. Even with identical Python code, the underlying hardware and runtime execute the math differently. The A100 offline evaluation likely used PyTorch with TF32 (TensorFloat-32) enabled by default, which truncates the mantissa to 10 bits. The production T4 (which doesn't support TF32) might be running a TensorRT compiled engine in FP16, or falling back to FP32. Furthermore, different CUDA versions or cuDNN libraries between the clusters can select different convolution algorithms (e.g., Winograd vs. GEMM) which accumulate floating-point rounding errors differently. Over a 50-layer network, these micro-differences compound into completely different activation distributions at the final classification layer.
Napkin Math: A single FP32 vs FP16 addition can have a relative error of $\sim 10^{-4}$. In a ResNet-50, an input image undergoes $\sim 10^9$ MAC operations. If the A100 uses TF32 (10-bit mantissa) and the T4 uses FP16 (10-bit mantissa) but with different rounding modes or accumulation orders in the Tensor Core, the final logit values can drift by 5-10%. For edge-case inputs where the model is uncertain (logits: [0.51, 0.49]), this hardware-level numerical drift is enough to flip the argmax classification, dropping overall accuracy from 95% to 70%.
π Deep Dive: ML Operations
</details>Interviewer: "Our model degrades by 1% accuracy every week. Retraining the model costs us $50,000 in GPU time. A 1% accuracy drop costs the business $100,000 a week. Exactly how often should we trigger a retraining pipeline?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Retrain weekly since the cost of degradation ($100k) exceeds the cost of retraining ($50k)." This is directionally right but not optimal.
Realistic Solution: Cost-aware automation. You do not retrain based on a calendar; you retrain when the cumulative cost of performance degradation intersects with the fixed cost of retraining. This is the Retraining Staleness Model. You must formulate the threshold mathematically so the MLOps pipeline triggers training autonomously when the financial math dictates it.
Napkin Math: Degradation cost accumulates: week 1 = $100k, week 2 = $200k cumulative, week 3 = $300k. Retraining costs $50k. Optimal retrain point: when cumulative degradation cost = retraining cost. $\sum_{i=1}^{n} 100k \times i% > 50k$ β retrain roughly every $\sqrt{2 \times 50k / 100k} \approx 1$ week. But if retraining cost were $500k, optimal interval stretches to ~3 weeks.
π Deep Dive: ML Operations
</details>Interviewer: "We want to shadow-test a new 70B LLM before replacing our production 13B model. The plan: mirror all production traffic to the new model, compare outputs, then swap. The infrastructure team comes back and says the shadow deployment will cost more than the production deployment itself. Why is shadow-testing an LLM so much more expensive than shadow-testing a traditional ML model, and how do you make it feasible?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Shadow deployment is free β we're just logging predictions." This is true for a 100ms classification model. It's catastrophically wrong for autoregressive LLMs.
Realistic Solution: Shadow-testing a 70B LLM is expensive because you must run full autoregressive decoding for every request β not just a single forward pass. A classification model shadow costs one forward pass (~10ms, ~0.1 GPU-second). An LLM shadow costs prefill + N decode steps (~2-5 seconds of GPU time per request). At 1,000 QPS production traffic, the shadow needs 2,000-5,000 GPU-seconds per second of wall time β more GPUs than production itself. The fix: sample shadow traffic (10-20% of requests), use speculative decoding with the production 13B as the draft model, or run shadow only on prefill (compare logits at the first token without full generation).
Napkin Math: Production 13B at 1,000 QPS: ~50 A100s (continuous batching, 20 requests/GPU). Shadow 70B at 1,000 QPS: needs ~200 A100s (5Γ more params, 4Γ less efficient batching). Shadow at 10% sampling: 20 A100s β now feasible. Prefill-only shadow (no decode): 5 A100s β cheap enough to run 100%.
π Deep Dive: ML Operations
</details>Interviewer: "We are purchasing a $100M cluster of H100 GPUs. The CFO wants to know the Total Cost of Ownership (TCO) over a 3-year lifecycle. Why is the $100M figure severely underestimating the budget?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Add 20% for networking and storage." Infrastructure costs matter, but the biggest hidden cost is ongoing.
Realistic Solution: You forgot the OpEx (Operating Expenses), specifically power and cooling. Over a 3-year lifespan, the electricity required to run a massive cluster at high utilization β plus the power to cool it β frequently matches or exceeds the initial CapEx (hardware cost). System efficiency ($\eta$) isn't just about training speed; it's the primary lever for financial viability.
Napkin Math: 10,000 H100s Γ 700W TDP = 7 MW compute. With PUE of 1.3 (cooling overhead): 9.1 MW total. At $0.10/kWh: $9,100/hour = $79.7M/year = $239M over 3 years in electricity alone. That's 2.4Γ the hardware cost. True TCO β $100M (CapEx) + $239M (power) + networking/staff = $350M+.
Key Equation: $\text{TCO} = \text{CapEx} + (\text{Power} \times \text{PUE} \times \text{Rate} \times \text{Hours}) + \text{Staff} + \text{Network}$
π Deep Dive: Sustainable AI
</details>Interviewer: "After a prompt injection incident, your team adds a safety classifier that screens every LLM input and output. The classifier is a fine-tuned BERT model that takes 15ms per call. Your LLM's TTFT was 200ms and TPOT was 25ms. Users now complain the system feels noticeably slower. The PM says '15ms is nothing.' Why is the PM wrong, and how do you fix it without removing the guardrail?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "15ms is only 7.5% of the 200ms TTFT β users won't notice." This ignores where the 15ms hits in the pipeline.
Realistic Solution: The guardrail runs twice per request (input screening + output screening) and the output screen runs on every generated token in streaming mode, not just once. Input screen adds 15ms to TTFT (200ms β 215ms β barely noticeable). But output screening adds 15ms per token chunk, turning TPOT from 25ms to 40ms β a 60% increase in perceived generation speed. For a 200-token response, total time goes from $200 + (200 \times 25) = 5.2s$ to $215 + (200 \times 40) = 8.2s$ β a 58% regression. The fix: batch output screening (check every 10 tokens instead of every token), run the classifier on a dedicated GPU so it doesn't contend with the LLM's KV-cache memory, or use a smaller distilled classifier (2ms instead of 15ms).
Napkin Math: Per-token screening at 25ms TPOT + 15ms guardrail = 40ms effective TPOT. At 200 tokens: 8.0s decode vs 5.0s without guardrail. Batch-10 screening: 25ms TPOT + 1.5ms amortized = 26.5ms effective TPOT. At 200 tokens: 5.3s decode β nearly invisible overhead.
π Deep Dive: Security and Privacy
</details>Interviewer: "Your medical imaging model achieves 94% accuracy across all demographic groups in FP32. You quantize to INT8 for deployment on edge devices in rural clinics. Overall accuracy drops to 92% β acceptable. But a post-deployment audit reveals accuracy for dark-skinned patients dropped from 91% to 72%, while light-skinned patients only dropped from 96% to 94%. How did quantization amplify a bias that barely existed in the original model?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Quantization is a uniform operation β it should degrade all subgroups equally." This assumes all subgroups occupy the same region of the activation space.
Realistic Solution: Quantization maps continuous activations to discrete bins. The bin boundaries are set by calibration data, which is dominated by the majority subgroup. Features that distinguish dark-skinned patients (subtle contrast differences, different melanin-related color distributions) occupy a narrow, low-magnitude region of the activation space. When you quantize, these fine-grained distinctions get crushed into the same bin β the model literally can't tell them apart anymore. Meanwhile, high-contrast features (dominant in light-skinned patients) span multiple bins and survive quantization. The fix: per-subgroup calibration data, mixed-precision (keep sensitive layers in FP16), or quantization-aware training with subgroup-balanced batches.
Napkin Math: FP32 has ~7 decimal digits of precision. INT8 has 256 levels. If dark-skin features cluster in a 0.01-wide activation range and INT8 bin width is 0.02, those features collapse to 1 bin (binary output). Light-skin features spanning a 0.5-wide range get 25 bins β plenty of resolution. The quantization error is 25Γ worse for the minority subgroup.
Key Equation: $\text{Quantization Error}{\text{subgroup}} \propto \frac{\text{Activation Range}{\text{subgroup}}}{\text{Number of INT8 Bins in Range}}$
π Deep Dive: Responsible Engineering
</details>Interviewer: "A security audit reveals our medical LLM is vulnerable to membership inference attacks β an attacker can determine if a specific patient's records were in the training set. The privacy team mandates Differentially Private SGD (DP-SGD) with Ξ΅=1. Your training infrastructure team comes back and says: 'DP-SGD will increase training time by 10Γ and require 3Γ more GPU memory.' Why does adding privacy guarantees have such a devastating systems cost, and how do you bring it down to something feasible?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "DP-SGD just adds noise to gradients β that should be nearly free." This ignores the per-example gradient requirement.
Realistic Solution: Standard SGD computes one gradient for the entire batch (efficient β one backward pass). DP-SGD requires per-example gradients because you must clip each example's gradient independently before aggregating. On an H100 with batch size 64, this means 64 separate backward passes instead of 1 β a 64Γ compute increase for the backward pass. The memory cost comes from storing 64 separate gradient tensors simultaneously. For a 7B model: standard gradient = 14 GB (FP16). Per-example gradients for batch 64 = 14 GB Γ 64 = 896 GB β doesn't fit on any single GPU. Fixes: (1) use JAX's vmap for efficient per-example gradients (2-3Γ overhead instead of 64Γ), (2) gradient accumulation with micro-batches of 1, (3) DP-LoRA (apply DP only to low-rank adapter weights, reducing the gradient tensor from 7B to ~10M parameters).
Napkin Math: 7B model, batch 64, FP16. Standard backward: 1 pass Γ 14 GB gradients = 14 GB. DP-SGD naive: 64 passes Γ 14 GB = 896 GB (impossible). DP-SGD with
vmap: 1 pass Γ 14 GB Γ ~3Γ overhead = 42 GB (fits on H100 80 GB). DP-LoRA (rank 16): 64 Γ 20 MB = 1.3 GB per-example gradients β trivial. Training time: standard = 1,000 GPU-hours. DP-SGD naive = 10,000 GPU-hours. DP-LoRA = 1,500 GPU-hours.
Key Equation: $\tilde{g} = \frac{1}{B}\sum_{i} \text{clip}(g_i, C) + \mathcal{N}(0, \sigma^2 C^2 I)$ β the per-example $g_i$ is what makes it expensive
π Deep Dive: Security and Privacy
</details>Interviewer: "You rent an 8x A100 node on AWS to train a ResNet on a 50TB dataset of high-resolution medical images. To save money, you leave the dataset in a standard S3 bucket and use PyTorch's IterableDataset to stream the images directly to the GPUs during training. Your GPUs sit at 5% utilization. Why did streaming from object storage starve your compute?"
Common Mistake: "Thinking of S3 as just another hard drive, ignoring the massive per-request latency and limited network bandwidth compared to local NVMe."
Realistic Solution: You hit the Data Wall. A GPU is a math engine that must be fed. An 8x A100 node can process thousands of images per second. However, pulling individual images from S3 introduces massive HTTP network latency (Time To First Byte is ~50-100ms per request). Furthermore, the total inbound network bandwidth of the EC2 instance (e.g., 25 Gbps) is physically incapable of keeping up with the VRAM consumption rate of 8 A100s. The GPUs are spending 95% of their time waiting for the network card.
Napkin Math: 8x A100s training ResNet-50 can process ~20,000 images/sec. If each medical image is 2MB, the GPUs demand
20,000 * 2MB = 40 GB/s(320 Gbps) of constant data flow. A standard cloud instance might only have a 25 Gbps (~3 GB/s) network link to S3. Even with perfect pipelining, the network pipe is physically 10x too small to feed the GPUs. You must copy the dataset to local RAID-0 NVMe drives first.
π Deep Dive: Data Engineering
</details>Interviewer: "Your team builds a feature store to eliminate training-serving skew. The store materializes 2,000 features into an online Redis cluster and an offline Parquet lake. After launch, you discover that 15% of features have different values between the online and offline paths for the same entity at the same timestamp. Your feature store was supposed to solve this exact problem. What went wrong?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "There's a bug in the ETL pipeline" or "Redis and Parquet use different serialization." Both assume a simple data bug β the real issue is architectural: time semantics.
Realistic Solution: The online path computes features at request time using the latest state (point-in-time lookup), while the offline path computes features in a batch job that runs hours later using event-time windows. The divergence comes from late-arriving data. A user's "last-7-day purchase count" computed online at 2:00 PM sees transactions up to 1:59 PM. The offline batch job running at midnight replays the same window but now includes transactions that arrived late (payment processor delays, retry queues). The fix is a dual-write architecture: compute features once in a streaming engine (Flink/Spark Streaming), write simultaneously to both online (Redis) and offline (Parquet) stores from the same computation. This guarantees bitwise consistency. Additionally, implement a feature validation pipeline that samples 1% of online-served features, replays them offline, and alerts on divergence >0.1%.
Napkin Math: 2,000 features Γ 10M entities = 20B feature values. Redis cluster for online serving: 20B Γ 8 bytes avg = 160 GB β 3 nodes of r6g.4xlarge (128 GB each) at $1.60/hr = $3,456/month. Parquet offline: 160 GB compressed (~40 GB) on S3 = $0.92/month. The cost asymmetry is 3,750:1, which is why teams are tempted to compute them differently β and that's where skew creeps in. Streaming dual-write adds ~$800/month in Flink compute but eliminates the 15% skew that was causing a 3% model accuracy drop worth $200k/month in revenue.
π Deep Dive: Data Engineering
</details>Interviewer: "You're deploying a fine-tuned 70B LLM to replace your production 13B model. You set up a standard canary: route 5% of traffic to the new model, monitor error rate and latency for 30 minutes, then promote. After full rollout, users report the new model occasionally hallucinates wildly long, repetitive outputs. Your canary detected no latency or error spikes. Why do standard infra metrics miss LLM quality regressions, and how could you have used the GPU's KV-cache memory profile as a hardware-level canary signal?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "The canary window was too short β extend it to 24 hours." More time helps, but the fundamental problem is that LLM failures are not captured by traditional canary metrics.
Realistic Solution: Traditional canary metrics (latency, error rate, throughput) measure infrastructure health, not output quality. A model that suddenly starts outputting repetitive garbage ("I am an AI, I am an AI...") will still return HTTP 200s and keep the GPU busy. However, LLM quality regressions often manifest as changes in generation length (verbosity or truncation). You can use the GPU's KV-cache memory allocation as a proxy metric for output behavior. If the new model is hallucinating longer responses, the average KV-cache blocks allocated per request will spike. If it's failing to reason and returning short "I don't know" answers, the KV-cache allocation will drop. Monitoring the physical memory footprint provides a real-time, hardware-level signal of behavioral changes before users complain.
Napkin Math: Baseline model generates 200 tokens/request average. Canary model has a repetition bug and generates 800 tokens/request. Latency might look okay if the batch scheduler just drops throughput to compensate. But KV-cache memory tells the truth: Baseline uses 200 tokens Γ 1MB/token (for a large batch) = 200MB per request. Canary uses 800MB per request. By monitoring
vllm:gpu_cache_usage_perc, you would see the canary instances hitting 90% KV-cache saturation 4Γ faster than the baseline instances, instantly flagging a behavioral shift.
π Deep Dive: ML Operations
</details>Interviewer: "Your real-time ML pipeline ingests clickstream events from Kafka (50,000 events/sec), enriches them with user features from Redis, runs a fraud scoring model, and writes results to a downstream Kafka topic. During a flash sale, event rate spikes to 200,000/sec. Within 90 seconds, the fraud model's latency jumps from 5ms to 800ms, even though GPU utilization is only 40%. What is happening, and how do you prevent it?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "The GPU is the bottleneck β scale up the inference cluster." GPU utilization is 40%, so the GPU is not the problem.
Realistic Solution: This is a backpressure cascade. The bottleneck is the Redis feature lookup, not the GPU. At 50k/sec, each Redis lookup takes ~1ms (well within budget). At 200k/sec, the Redis cluster hits its throughput ceiling (~150k ops/sec on a 3-node cluster), and lookup latency spikes to 50-100ms. Events queue up in the enrichment stage's in-memory buffer. The buffer grows until it triggers GC pauses in the JVM/Python runtime, which stalls the Kafka consumer. Kafka's consumer group rebalances because the consumer appears dead (no heartbeat during GC). Rebalancing causes all consumers in the group to pause for 30-60 seconds, creating a massive backlog. When consumers resume, they replay the backlog, overwhelming Redis again β a feedback loop. The GPU sits idle because it never receives batches to score. The fix: (1) implement backpressure signaling β when the enrichment queue exceeds a threshold, shed load by sampling events (score every 4th event during spikes), (2) add a circuit breaker on Redis lookups (fall back to cached features after 10ms timeout), (3) use Kafka's max.poll.records to cap batch sizes and prevent consumer timeout.
Napkin Math: Normal: 50k events/sec Γ 1ms Redis = 50 concurrent Redis connections (fine for 3-node cluster). Spike: 200k/sec Γ 1ms = 200 concurrent connections β Redis saturates at 150k, queue builds at 50k/sec. In 30 seconds: 1.5M events queued Γ ~200 bytes each = 300 MB in-memory buffer β JVM GC pause. Consumer heartbeat timeout = 10s β rebalance takes 45s β 200k Γ 45 = 9M event backlog. Recovery time at 150k/sec drain rate: 9M / 150k = 60 seconds of catch-up, during which new events continue arriving.
π Deep Dive: Data Engineering
</details>Interviewer: "Your team has assembled a 10TB pre-training dataset by crawling the web. A data quality audit reveals approximately 30% near-duplicate content. Your ML lead says 'duplicates don't matter β more data is always better.' Your infrastructure lead says 'dedup everything β it'll save 30% on training compute.' Who is right, and what is the economically optimal deduplication strategy?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Deduplicate everything to save compute" or "Keep all duplicates because more data helps." Both are wrong β the relationship between duplication and model quality is non-linear.
Realistic Solution: Neither is right. Research (Lee et al., 2022; Penedo et al., 2023) shows that moderate deduplication improves model quality (reduces memorization, improves generalization), but aggressive deduplication hurts it (removes legitimate repeated patterns like common phrases, code idioms). The optimal strategy is fuzzy deduplication with a similarity threshold: use MinHash LSH to identify near-duplicate clusters (Jaccard similarity >0.8), keep one representative from each cluster, but preserve exact duplicates below the threshold. The economics: deduplication itself is not free β running MinHash on 10TB requires significant compute. The decision framework is: compare the cost of deduplication compute against the savings in training compute, weighted by the quality impact.
Napkin Math: 10TB dataset, 30% near-duplicates. Training a 7B model on 10TB: ~3,000 A100-hours at $2/hr = $6,000. Training on 7TB (after dedup): ~2,100 A100-hours = $4,200. Savings: $1,800 per training run. Deduplication cost: MinHash LSH on 10TB β 200 CPU-hours at $0.05/hr = $10 (one-time). But the quality impact matters more: deduped models show 2-5% lower perplexity, which compounds across downstream tasks. If you retrain monthly, annual savings = $1,800 Γ 12 + quality gains = $21,600 + downstream value. The dedup ROI is ~2,160:1 on pure compute savings alone.
π Deep Dive: Data Engineering
</details>Interviewer: "You manage a shared GPU cluster: 256 A100 80GB GPUs across 32 nodes (8 GPUs per node, NVLink within node, InfiniBand between nodes). Three teams submit jobs simultaneously: Team A wants 64 GPUs for a 70B model training run (needs NVLink topology β all 8 GPUs per node), Team B wants 128 GPUs for a distributed data-parallel job (topology-flexible), Team C wants 32 GPUs for hyperparameter sweeps (many small 1-GPU jobs). How do you schedule these to maximize cluster utilization without starving any team?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "First-come-first-served" or "Give each team their fair share (85 GPUs each)." FCFS leads to fragmentation and starvation. Equal shares ignore topology constraints.
Realistic Solution: This requires topology-aware gang scheduling with preemption policies. Team A's 70B training requires tensor parallelism across NVLink-connected GPUs β it must get full 8-GPU nodes (gang scheduling). Allocate 8 nodes (64 GPUs) as a contiguous NVLink-connected block. Team B's data-parallel job is topology-flexible β each replica just needs 1 GPU, and gradient sync over InfiniBand is acceptable. Allocate 16 nodes (128 GPUs), but these can be any available nodes. Team C's hyperparameter sweeps are embarrassingly parallel 1-GPU jobs β they can fill any gaps. Allocate the remaining 8 nodes (64 GPUs), but also allow Team C to backfill into any GPU left idle by Teams A or B during checkpointing, data loading, or communication stalls. Implement a priority system: Team A gets guaranteed NVLink nodes (highest topology constraint), Team B gets guaranteed GPU count (medium constraint), Team C gets best-effort backfill (lowest constraint but highest flexibility). Use Kubernetes with the NVIDIA GPU Operator and a custom scheduler plugin that understands NVLink topology.
Napkin Math: Without topology awareness: Team A gets 64 GPUs across 12 nodes (some nodes split) β NVLink broken β tensor parallel falls back to PCIe β 3.5Γ slower training. With topology-aware scheduling: Team A gets 8 full nodes β NVLink bandwidth 600 GB/s per node vs 64 GB/s PCIe β training runs at full speed. Cluster utilization without backfill: (64 + 128 + 64) / 256 = 100% allocated, but actual GPU utilization ~65% (idle during checkpoints, data loading). With Team C backfill: utilization rises to ~82%. Annual savings at $2/GPU-hr: 256 GPUs Γ 8,760 hrs Γ 17% improvement Γ $2 = $764k/year.
π Deep Dive: ML Operations
</details>Interviewer: "Your team trains a 7B parameter model that takes 72 hours on 8 A100 GPUs. On-demand pricing is $2.00/GPU-hour. Spot instances are $0.60/GPU-hour β a 70% discount. Your manager says 'use spot instances and save $806.' Midway through a 72-hour run, your spot instances get preempted. What is the true expected cost of spot training, and when does on-demand actually become cheaper?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Spot is always cheaper because the hourly rate is 70% less." This ignores the cost of preemption: lost compute, checkpoint overhead, and extended wall-clock time.
Realistic Solution: Spot instance economics depend on three factors: (1) preemption probability β A100 spot instances in popular regions have ~15-25% chance of preemption per hour, (2) checkpoint overhead β you must checkpoint frequently enough that preemption doesn't lose too much work, but checkpointing itself costs time and storage, (3) restart overhead β after preemption, you wait for new instances (5-30 min), reload the checkpoint, and warm up the data pipeline. The expected cost model: if preemption probability per hour is p=0.20, expected number of preemptions in 72 hours β 72 Γ 0.20 = 14.4 preemptions. Each preemption wastes the work since the last checkpoint plus restart time. With checkpoints every 30 minutes and 15-minute restart overhead, each preemption wastes ~45 minutes of 8-GPU time. Expected wasted compute: 14.4 Γ 0.75 hrs Γ 8 GPUs = 86.4 GPU-hours. Total spot cost: (72 + 86.4 Γ 72/576) Γ 8 Γ $0.60 + storage = ~$403. On-demand: 72 Γ 8 Γ $2.00 = $1,152. Spot still wins here, but the breakeven is when preemption rate exceeds ~40%/hr or checkpoint frequency is too low.
Napkin Math: On-demand: 72 hrs Γ 8 GPUs Γ $2.00 = $1,152. Spot (optimistic, 10% preemption/hr): ~5 preemptions Γ 45 min wasted = 3.75 hrs extra β 75.75 hrs Γ 8 Γ $0.60 = $364. Spot (pessimistic, 30% preemption/hr): ~22 preemptions Γ 45 min = 16.5 hrs extra β 88.5 hrs Γ 8 Γ $0.60 = $425, plus checkpoint storage: 22 checkpoints Γ 14 GB Γ $0.023/GB = $7. Breakeven: spot becomes more expensive than on-demand when effective hours exceed 1,152 / (8 Γ $0.60) = 240 hours β meaning preemptions add >168 hours of overhead (unlikely but possible with very high preemption rates and no checkpointing).
π Deep Dive: Sustainable AI
</details>Interviewer: "Your company pledges net-zero AI training by 2027. You operate GPU clusters in three regions: Virginia (avg 380 gCOβ/kWh, coal-heavy grid), Oregon (avg 80 gCOβ/kWh, hydro-heavy), and Iceland (avg 28 gCOβ/kWh, geothermal). A 70B model training run requires 1,000 A100-hours regardless of location. Your CFO says 'just buy carbon offsets β it's cheaper than moving workloads.' Design a carbon-aware training scheduler and prove the CFO wrong (or right) with numbers."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Run everything in Iceland β lowest carbon." This ignores that carbon intensity varies by time of day (solar/wind availability), that data gravity makes cross-region training expensive, and that Iceland has limited GPU capacity.
Realistic Solution: A carbon-aware scheduler must optimize across three dimensions: (1) spatial shifting β route jobs to the lowest-carbon region with available capacity, (2) temporal shifting β delay non-urgent jobs to times when the grid is cleanest (midday solar peaks, windy nights), (3) workload shaping β run compute-intensive phases (forward/backward pass) during clean periods and I/O-intensive phases (checkpointing, data loading) during dirty periods. The scheduler ingests real-time grid carbon intensity from APIs (WattTime, ElectricityMaps), maintains a job priority queue, and makes placement decisions every 15 minutes. For the 70B training run: use Oregon as primary (80 gCOβ/kWh), with temporal shifting to avoid the 6-9 PM peak (when gas peakers fire up, pushing intensity to 200 gCOβ/kWh). Cross-region data transfer for training is expensive (100TB dataset over 10 Gbps = 22 hours), so the scheduler must factor in the carbon cost of data movement.
Napkin Math: 1,000 A100-hours Γ 700W = 700 kWh. Virginia: 700 Γ 380 = 266 kgCOβ. Oregon: 700 Γ 80 = 56 kgCOβ. Iceland: 700 Γ 28 = 19.6 kgCOβ. Carbon offset cost: $50-100/tonne β Virginia offset = 0.266t Γ $75 = $20. Oregon compute premium over Virginia: ~$0.30/GPU-hr Γ 1,000 = $300. Iceland premium: ~$0.80/GPU-hr Γ 1,000 = $800. The CFO is right that offsets ($20) are cheaper than relocation ($300-800) β if you trust offsets. But regulatory trend (EU CSRD, SEC climate rules) is toward Scope 2 market-based accounting where offsets don't count. Under those rules, Virginia training is a 266 kgCOβ liability regardless of offsets. Temporal shifting in Oregon (avoiding peak hours) reduces effective intensity from 80 to ~50 gCOβ/kWh at zero additional cost β 700 Γ 50 = 35 kgCOβ, a 37% reduction for free.
π Deep Dive: Sustainable AI
</details>Interviewer: "Your platform serves 5 different LLMs on a shared pool of 64 A100 GPUs using vLLM with continuous batching. The finance team wants to charge each product team for their inference costs. Your colleague proposes: 'Divide total GPU cost by total requests, charge per request.' The search team (short prompts, 50-token outputs) and the content team (long prompts, 2,000-token outputs) both object. Design a fair cost attribution model."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Charge per request" or "Charge per token." Per-request ignores the 40Γ difference in compute between a 50-token and 2,000-token generation. Per-token ignores that prefill and decode have different costs.
Realistic Solution: Fair cost attribution must reflect actual GPU resource consumption, which has two distinct phases: (1) Prefill cost β proportional to input tokens (compute-bound, scales as O(nΒ²) for attention), and (2) Decode cost β proportional to output tokens (memory-bandwidth-bound, scales linearly but occupies KV-cache memory for the entire generation duration). The correct model charges: cost = Ξ± Γ input_tokens + Ξ² Γ output_tokens + Ξ³ Γ generation_time_ms. The Ξ± term captures prefill compute, Ξ² captures decode compute, and Ξ³ captures KV-cache memory occupancy (a long-running generation blocks memory that could serve other requests). Calibrate Ξ±, Ξ², Ξ³ by profiling each model: measure GPU-seconds consumed per input token (prefill) and per output token (decode) at representative batch sizes. For continuous batching, the Ξ³ term is critical β a request generating 2,000 tokens at 40ms/token occupies KV-cache for 80 seconds, blocking ~2 GB of GPU memory that could serve 20 short requests.
Napkin Math: 64 A100s at $2/hr = $128/hr total. Search team: 10,000 req/hr Γ 500 input + 50 output tokens. Content team: 1,000 req/hr Γ 2,000 input + 2,000 output tokens. Per-request billing: search pays 10/11 Γ $128 = $116/hr, content pays $12/hr. Per-token billing: search = 5.5M tokens, content = 4M tokens β search pays $74, content pays $54. Per-GPU-second billing (actual resource use): search prefill β 0.5s/req Γ 10k = 5,000 GPU-sec, content prefill β 4s/req Γ 1k = 4,000 GPU-sec, content decode β 80s/req Γ 1k = 80,000 GPU-sec. Content uses 94% of decode resources β fair charge β $108/hr. The naive per-request model undercharges content by 9Γ.
π Deep Dive: ML Operations
</details>Interviewer: "Your platform has 23 models in production, some dating back 3 years. You discover that 8 older models collectively serve only 2% of total traffic but consume 30% of your GPU fleet. Your PM asks why these old models can't just be co-located on the same GPUs as the newer models to save money. Why does maintaining multiple model generations simultaneously fragment your serving cluster and destroy GPU utilization?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just put them all in the same Docker container." This ignores how GPUs allocate memory and schedule execution for heterogeneous ML workloads.
Realistic Solution: Co-locating multiple model generations on a single GPU creates severe memory fragmentation and scheduling bottlenecks. Old models often use different precision formats (e.g., FP32 vs FP16), different CUDA graph compilations, and different batch sizes. When you load multiple distinct models onto one GPU, each model statically reserves its own CUDA context, workspace memory, and KV-cache blocks. Because the models don't share weights or attention architectures, the serving framework (like vLLM or Triton) cannot easily batch requests across them. The GPU ends up context-switching between different kernels, destroying arithmetic intensity, while the VRAM is partitioned into small, inflexible silos that prevent any single model from achieving a large enough batch size to be compute-bound.
Napkin Math: An 80GB H100 running one 13B model (26GB weights) has 54GB available for a massive, unified KV-cache, allowing continuous batching of 100+ concurrent requests (high utilization). If you co-locate three different 7B models (14GB weights each = 42GB total), you only have 38GB left. Worse, that 38GB is fragmented across three separate KV-cache pools (12.6GB each). Each model can now only batch ~20 requests. The GPU spends its time context-switching between the three models' kernels, operating at low batch sizes, dropping effective throughput by 3-4Γ compared to serving a single model.
π Deep Dive: ML Operations
</details>Interviewer: "Your research team trains a model that achieves state-of-the-art results. When the production team retrains with the 'same' code and data, accuracy is 3% lower. They try 5 more times β each run produces a different result, varying by up to 2%. The research team insists their code is deterministic. Where are the hidden sources of non-determinism, and what does it cost to eliminate them on a modern GPU cluster?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Set the random seed and it's reproducible." Seeds control Python/NumPy/PyTorch RNG, but GPUs have hardware-level non-determinism that seeds cannot fix.
Realistic Solution: There are at least five layers of non-determinism in GPU training: (1) cuDNN autotuning β cuDNN benchmarks multiple kernel implementations at startup and picks the fastest; different runs may select different kernels with different numerical properties. Fix: torch.backends.cudnn.deterministic = True (5-15% slower). (2) Atomic floating-point reductions β operations like scatter_add, batch norm, and attention use atomic adds on GPU, which are non-associative in floating point; the order depends on thread scheduling. Fix: use deterministic algorithms (torch.use_deterministic_algorithms(True)), which disables some fast kernels. (3) Multi-GPU gradient reduction β NCCL AllReduce order varies with network timing; FP16 gradient summation in different orders produces different results. Fix: enforce a fixed reduction tree (slower). (4) Data loading order β multi-worker DataLoader with shuffle=True produces different orderings even with the same seed if worker count or prefetch factor changes. Fix: use a deterministic sampler with explicit seed. (5) Hardware variation β different GPU silicon (even same model) has slightly different FP rounding behavior. Fix: impossible without emulation.
Napkin Math: Performance cost of full determinism on 8Γ A100 training: cuDNN deterministic mode: -10% throughput. Deterministic algorithms: -15% (disables fast atomics). Fixed NCCL reduction: -5% (serialized communication). Total: ~30% slower training. For a 72-hour training run: 72 / 0.70 = 103 hours deterministic. Extra cost: 31 hrs Γ 8 GPUs Γ $2/hr = $496 per run. If you need 5 reproducible runs for a paper: $2,480 in determinism tax. Most teams accept Β±1% variance and run 3 seeds instead: 3 Γ 72 hrs Γ 8 Γ $2 = $3,456 β more expensive but gives confidence intervals rather than false precision.
π Deep Dive: ML Operations
</details>Interviewer: "Your team wants to load a 13B-parameter LLM onto a single GPU for inference. The GPU has 24 GB of VRAM. They plan to load the model in FP16. Will it fit? Show your math."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "13 billion parameters Γ 2 bytes = 26 GB, so it won't fit β we need a 32 GB card." This gets the weight math right but stops there. The real mistake is thinking weights are the only memory consumer.
Realistic Solution: Each parameter in FP16 occupies 2 bytes. 13B Γ 2 = 26 GB for weights alone β already over the 24 GB budget. But the real picture is worse: you also need memory for the KV-cache, activation buffers, and the CUDA context (~500 MBβ1 GB). Even on a 32 GB card, the KV-cache for a single request at 4k context eats another ~1β2 GB, leaving almost no room for batching. The practical answer: you need either a 48 GB+ card (A6000, L40S), quantization to INT8 (13 GB weights) or INT4 (6.5 GB weights), or tensor parallelism across two 24 GB cards.
Napkin Math: FP16 weights: 13B Γ 2 bytes = 26 GB. INT8 weights: 13B Γ 1 byte = 13 GB. INT4 weights: 13B Γ 0.5 bytes = 6.5 GB. CUDA context overhead: ~0.8 GB. KV-cache at 4k context (40 layers, 40 heads, dim 128, FP16): $2 \times 40 \times 40 \times 128 \times 4096 \times 2 \approx 3.4$ GB per request. On a 24 GB card with INT8: 13 + 0.8 + 3.4 = 17.2 GB β fits with ~7 GB headroom for batching. On a 24 GB card with FP16: doesn't even load.
π Deep Dive: Hardware Acceleration
</details>Interviewer: "You're deploying a 70B-parameter LLM on a single H100 (80 GB HBM3, 3.35 TB/s bandwidth). During autoregressive decoding, roughly how many tokens per second can you generate for a single request? Show why this is a memory-bandwidth problem, not a compute problem."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "The H100 has 989 TFLOPS, so we can do a lot of compute per token β throughput should be thousands of tokens per second." This confuses compute capacity with the actual bottleneck.
Realistic Solution: During autoregressive decoding, each token requires reading the entire model weights from HBM once (the batch size is 1, so there's no reuse). The arithmetic intensity is ~1 Op/Byte β deep in the memory-bandwidth-bound regime. Throughput is dictated entirely by how fast you can stream weights through the memory bus, not by how many FLOPS the tensor cores can do.
Napkin Math: 70B params in FP16 = 140 GB. Each decode step reads all weights once. H100 bandwidth = 3.35 TB/s. Time per token = 140 GB / 3,350 GB/s β 42 ms. Throughput β 1000 / 42 β 24 tokens/sec for a single request. The tensor cores are doing 2 Γ 70B = 140 GFLOPS per token β that's 0.014% utilization of 989 TFLOPS. The GPU is 99.99% idle on compute, 100% saturated on bandwidth. Batching helps: with batch=8, you read weights once and reuse across 8 requests, pushing effective throughput to ~190 tokens/sec total.
π Deep Dive: Hardware Acceleration
</details>Interviewer: "You're fine-tuning a 7B-parameter model in FP16. Your GPU has 24 GB of VRAM. The model weights are 14 GB. Your engineer says 'we have 10 GB left β plenty for training.' Why will this OOM immediately?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "We need memory for activations, but 10 GB should be enough for a small batch." This forgets the two largest memory consumers in training: gradients and optimizer states.
Realistic Solution: Training requires storing: (1) model weights, (2) gradients (same size as weights), and (3) optimizer states. With Adam, you store two additional copies β the first and second moment estimates β each the same size as the weights, in FP32 for numerical stability. The gradient + optimizer memory dwarfs the weights themselves.
Napkin Math: Weights (FP16): 7B Γ 2 bytes = 14 GB. Gradients (FP16): 7B Γ 2 bytes = 14 GB. Adam optimizer states (FP32): first moment (7B Γ 4 = 28 GB) + second moment (7B Γ 4 = 28 GB) = 56 GB. Master weights copy (FP32): 7B Γ 4 = 28 GB. Total: 14 + 14 + 56 + 28 = 112 GB β before a single activation is stored. On 24 GB? You need either LoRA (trains <1% of parameters), DeepSpeed ZeRO-3 (shards everything across GPUs), or gradient checkpointing + offloading.
π Deep Dive: Training
</details>Interviewer: "You have a 500 GB dataset of image-text pairs (100M samples). You're training a CLIP-style model on 8Γ A100 GPUs. Each GPU processes 256 samples/sec. How long will one epoch take? What if the data pipeline can only deliver 1,500 samples/sec?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "8 GPUs Γ 256 samples/sec = 2,048 samples/sec. 100M / 2,048 = ~13.5 hours per epoch." This assumes the data pipeline can keep up β it often can't.
Realistic Solution: The compute math gives a lower bound, but the actual throughput is $\min(\text{GPU throughput}, \text{data pipeline throughput})$. If your data loading, decoding, and preprocessing pipeline is bottlenecked at 1,500 samples/sec, the 8 GPUs are starved 27% of the time. The effective throughput is 1,500, not 2,048.
Napkin Math: Compute-limited: 100M / 2,048 = 48,828 sec β 13.6 hours. Data-pipeline-limited: 100M / 1,500 = 66,667 sec β 18.5 hours. That's 5 extra hours per epoch β for a 10-epoch run, 50 wasted hours Γ 8 GPUs Γ $2/hr = $800 wasted on idle GPUs. Fixes: NVIDIA DALI for GPU-accelerated decoding, NVMe staging instead of NFS, WebDataset sharded format for parallel I/O. A $200/month NVMe cache can save $800/run.
π Deep Dive: Data Engineering
</details>Interviewer: "You need to load a 7B model (14 GB in FP16) from CPU RAM to GPU VRAM over PCIe Gen4 x16. How long does the transfer take? Your serving SLA requires cold-start latency under 5 seconds. Will you meet it?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "PCIe Gen4 x16 is 32 GB/s, so 14 GB / 32 = 0.44 seconds β easily under 5 seconds." This uses the theoretical peak and ignores real-world overheads.
Realistic Solution: PCIe Gen4 x16 has a theoretical unidirectional bandwidth of ~32 GB/s, but effective throughput for large DMA transfers is typically 24β26 GB/s due to protocol overhead, TLP framing, and IOMMU translation. Additionally, model loading involves deserialization (unpacking safetensors/pickle), memory allocation on GPU, and CUDA context initialization β none of which are pure DMA.
Napkin Math: Pure DMA at 25 GB/s effective: 14 GB / 25 = 0.56 sec. But real model loading: deserialization from disk to CPU RAM (~2 sec for NVMe, ~8 sec for HDD), CPUβGPU DMA (0.56 sec), CUDA context + kernel warmup (0.5β1 sec). Total cold start: ~3β4 sec from NVMe, ~10 sec from HDD. NVMe meets the 5-sec SLA; HDD doesn't. PCIe Gen5 doubles bandwidth to ~50 GB/s effective, cutting the DMA portion to 0.28 sec β but the disk read and deserialization dominate anyway.
π Deep Dive: Hardware Acceleration
</details>Interviewer: "Your startup wants to pre-train a 70B-parameter LLM on 2 trillion tokens. You're budgeting for H100 GPU hours on a cloud provider at $3.50/GPU-hour. Estimate the total training cost. What's the biggest risk to your budget?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Use the Chinchilla scaling law to get total FLOPS, divide by H100 peak FLOPS, multiply by cost." This uses peak FLOPS and ignores Model FLOP Utilization (MFU), which is the single biggest variable in the cost estimate.
Realistic Solution: The standard approximation for transformer training FLOPS is $6 \times N \times D$ where $N$ = parameters and $D$ = tokens. The key variable is MFU β what fraction of the GPU's peak FLOPS you actually sustain. State-of-the-art distributed training achieves 30β50% MFU; poorly optimized setups hit 15β25%.
Napkin Math: Total FLOPS = $6 \times 70\text{B} \times 2\text{T} = 8.4 \times 10^{23}$ FLOPS. H100 BF16 peak = 989 TFLOPS. At 40% MFU: effective = 396 TFLOPS/GPU. GPU-seconds needed = $8.4 \times 10^{23} / (3.96 \times 10^{14}) = 2.12 \times 10^{9}$ sec. GPU-hours = 589,000. With 512 GPUs: 1,151 hours β 48 days. Cost = 589,000 Γ $3.50 = $2.06M. At 25% MFU (poor optimization): $3.3M β a 60% budget overrun. The biggest risk: MFU dropping due to communication overhead, data pipeline stalls, or checkpointing pauses. Every 5% MFU drop costs ~$250k.
π Deep Dive: Training
</details>Interviewer: "You're serving a 70B LLM with 128k context length on 8Γ H100 GPUs (80 GB each, 640 GB total). The model weights in FP16 are 140 GB. Your product manager asks 'how many concurrent 128k-context users can we serve?' Calculate the answer."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "640 GB total β 140 GB weights = 500 GB free. KV-cache is small, so we can serve hundreds of users." This dramatically underestimates KV-cache size at long contexts.
Realistic Solution: The KV-cache grows linearly with both sequence length and batch size. At 128k tokens, a single request's KV-cache for a 70B model is enormous β comparable to the model weights themselves. This is the fundamental reason long-context serving is so expensive.
Napkin Math: Llama-70B: 80 layers, 8 KV-heads (GQA), head_dim = 128. KV-cache per request at 128k tokens (FP16): $2 \times 80 \times 8 \times 128 \times 128{,}000 \times 2 \approx 41.9$ GB. Available VRAM: 640 β 140 (weights) β 8 (CUDA overhead) = 492 GB. Max concurrent 128k users: $\lfloor 492 / 41.9 \rfloor = 11$ users. That's 11 concurrent users on $200k+ of hardware. At 4k context, the same KV-cache is only 1.31 GB β 375 concurrent users. The 32Γ context increase causes a 34Γ drop in concurrency. This is why production systems use PagedAttention (vLLM), KV-cache quantization (FP8), and prompt caching.
π Deep Dive: Frameworks
</details>Interviewer: "You're building a preprocessing pipeline for a 100 TB web-crawl dataset to train a foundation model. The pipeline must: deduplicate, filter toxic content, extract text, tokenize, and shuffle. Your cluster has 128 CPU nodes (64 cores each) and 1 PB of storage. Design the pipeline and estimate end-to-end time."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Run a MapReduce job β it's just text processing." This ignores that deduplication at 100 TB scale requires global state (you can't deduplicate in a purely parallel map), and shuffling 100 TB requires careful I/O planning.
Realistic Solution: The pipeline must be staged because each step has different computational profiles. (1) Text extraction β embarrassingly parallel, CPU-bound. 128 nodes Γ 64 cores = 8,192 cores. At ~10 MB/sec per core: 100 TB / (8,192 Γ 10 MB/s) = ~1,250 sec β 21 min. (2) Deduplication β requires MinHash LSH with a global index. The index for 100 TB of text (~50B documents) needs ~200 GB of RAM for signatures. Use a distributed hash table across nodes. Pairwise comparison is $O(n)$ with LSH. At ~5 MB/sec per core: ~2,500 sec β 42 min. (3) Toxic content filtering β run a classifier on each document. If using a small BERT model on CPU: ~1,000 docs/sec per core. 50B docs / (8,192 Γ 1,000) = 6,100 sec β 1.7 hours. (4) Tokenization β fast, ~50 MB/sec per core: 100 TB / (8,192 Γ 50) = 250 sec β 4 min. (5) Global shuffle β I/O-bound. Must read and write 100 TB. At 2 GB/s per node (NVMe): 100 TB / (128 Γ 2 GB/s) = 400 sec β 7 min per pass, need 2 passes = 14 min.
Napkin Math: Total pipeline: 21 + 42 + 102 + 4 + 14 β 3 hours end-to-end on 128 nodes. The bottleneck is toxic content filtering (classifier inference). Optimization: use GPU-accelerated classifiers (8 GPUs can replace 128 CPU nodes for this step). Storage I/O: 100 TB read + 100 TB write per stage Γ 5 stages = 1 PB of I/O. At 256 GB/s aggregate cluster bandwidth: ~1 hour just in I/O. Real-world: plan for 6β8 hours including retries, stragglers, and I/O contention.
π Deep Dive: Data Engineering
</details>Interviewer: "You're responsible for evaluating a new LLM before production deployment. Your team proposes running it on 5 standard benchmarks (MMLU, HumanEval, etc.). The new model scores 5% higher than your current production model. Why might this evaluation be dangerously misleading from an infrastructure perspective, and how could deploying the 'better' model actually destroy your serving economics?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "If it scores well on benchmarks, it's ready for production." This conflates academic evaluation with production readiness and ignores the hardware cost of intelligence.
Realistic Solution: Standard benchmarks measure intelligence in a vacuum, ignoring the physical constraints of the serving cluster. A model might score 5% higher on MMLU but require a different architectural trade-off that wrecks your infrastructure. For example, the new model might achieve its score by using a larger vocabulary (increasing embedding table memory), removing Grouped Query Attention (GQA) in favor of Multi-Head Attention (MHA) which inflates the KV-cache by 8Γ, or simply being more verbose (generating 3Γ more tokens per response). A production evaluation framework must use latency-aware and memory-aware scoring: measuring not just accuracy, but TTFT, TPOT, and peak VRAM per request.
Napkin Math: Current model (Llama-2-13B with GQA): KV-cache for 2K tokens = ~300MB. New model (13B with MHA for "better reasoning"): KV-cache for 2K tokens = ~2.4GB. On an A100 (40GB) with 26GB weights, the current model can batch ~40 concurrent requests. The new "better" model can only batch ~5 requests before OOMing. To maintain the same QPS, you must scale your GPU cluster by 8Γ. That 5% MMLU bump just increased your monthly AWS bill from $50k to $400k. The evaluation was misleading because it didn't measure the hardware cost of the new architecture.
π Deep Dive: ML Operations
</details>Interviewer: "You're quantizing a 7B model from FP16 to INT4 (4-bit integers with group-wise scaling, group size 128). Your team says 'it's just rounding β the error is tiny.' Calculate the worst-case quantization error per group and explain when this error becomes catastrophic."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "INT4 has 16 levels, so the error is at most 1/16 of the range β negligible." This ignores how outlier weights interact with uniform quantization.
Realistic Solution: INT4 symmetric quantization maps the range $[-\text{absmax}, +\text{absmax}]$ to 16 levels (4-bit signed: -8 to +7). The step size is $\Delta = 2 \times \text{absmax} / 15$. The maximum rounding error per weight is $\Delta/2$. The problem: if a group of 128 weights has one outlier 10Γ larger than the rest, the step size is set by the outlier, and all other weights are quantized with a step size 10Γ too coarse. This is the outlier channel problem that makes naive INT4 fail on LLMs.
Napkin Math: Typical weight distribution: 99% of weights in $[-0.1, 0.1]$, but 1% outliers at $[-1.0, 1.0]$. Without outliers: $\Delta = 2 \times 0.1 / 15 = 0.013$, max error = 0.0067 β 6.7% relative error. With one outlier at 1.0: $\Delta = 2 \times 1.0 / 15 = 0.133$, max error = 0.067 β 67% relative error for the majority of weights. Group quantization (group=128) limits the blast radius: only 128 weights share one scale factor. GPTQ/AWQ solve this by reordering weights so outliers are isolated, or using mixed-precision (FP16 for outlier channels, INT4 for the rest). Compute impact: INT4 on H100 Tensor Cores = 1,979 TOPS vs 989 TFLOPS FP16 β 2Γ throughput if quantization error is controlled.
π Deep Dive: Model Compression
</details>Interviewer: "Your company runs 12 different LLMs in production (ranging from 1B to 70B parameters) serving 50,000 QPS total across all models. Currently each model has its own dedicated GPU pool. The CFO says GPU costs are $2M/month. Design a multi-model serving platform that cuts costs by at least 40% without violating per-model latency SLAs."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just put all models on the same GPUs and multiplex." Naive co-location causes memory fragmentation and SLA violations because large models evict small ones from VRAM.
Realistic Solution: A multi-model platform needs three architectural layers: (1) Model tiering β group models by size and latency SLA. Tier 1 (1Bβ7B, <100ms): always resident in VRAM, share GPUs via MPS or time-slicing. Tier 2 (13Bβ30B, <500ms): resident on dedicated GPUs with overflow to CPU offload. Tier 3 (70B, <2s): tensor-parallel across multi-GPU, dedicated but right-sized. (2) Predictive autoscaling β use traffic forecasting (not reactive scaling) to pre-warm models 10 minutes before demand spikes. (3) KV-cache sharing β for models serving similar prompts (e.g., system prompts), cache the KV-state of common prefixes across requests.
Napkin Math: Current: 12 models Γ dedicated pools. Typical utilization: 30% (peak-provisioned). Monthly cost: $2M. Optimized: Tier 1 (8 small models): consolidate from 40 GPUs to 15 (shared, 80% util). Savings: 25 GPUs Γ $2/hr Γ 720 = $36k/mo. Tier 2 (3 mid models): right-size from 60 GPUs to 35 with autoscaling. Savings: $36k/mo. Tier 3 (1 large model): reduce from 80 GPUs to 50 with continuous batching + PagedAttention. Savings: $43k/mo. Prefix caching saves ~20% of KV-cache memory across all tiers β another 15% GPU reduction: $60k/mo. Total savings: $175k/mo (8.75%) β wait, that's not 40%. The real lever: spot instances for Tier 1 (stateless, fast restart) saves 65% on those GPUs: $200k/mo. Plus quantizing Tier 2 to INT8 halves their GPU count: $180k/mo. Revised total: $850k/mo savings (42.5%).
π Deep Dive: ML Operations
</details>Interviewer: "You're serving a 70B LLM and must choose the tensor parallelism (TP) degree: TP=2, TP=4, or TP=8 across H100 GPUs connected via NVLink. Higher TP reduces per-GPU memory and per-token latency, but adds communication overhead. Calculate the optimal TP degree for a latency target of 40ms per output token."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Use TP=8 for maximum parallelism β more GPUs always means lower latency." This ignores that each TP step requires an AllReduce synchronization, and the communication cost grows with TP degree.
Realistic Solution: Each transformer layer with TP requires two AllReduce operations (one after the attention projection, one after the MLP). The AllReduce volume per operation is $2 \times (TP-1)/TP \times \text{hidden_size} \times \text{batch} \times \text{bytes}$. With NVLink at 900 GB/s bidirectional, the communication time per layer is small but multiplies across 80 layers and 2 AllReduces per layer.
Napkin Math: 70B model, 80 layers, hidden=8192, FP16, batch=1. Per-token compute per GPU: 140 GFLOP / TP. At 989 TFLOPS (but memory-bound, so use bandwidth): 140 GB weights / TP read at 3.35 TB/s per GPU. TP=2: Compute: 70 GB / 3.35 TB/s = 20.9 ms. Comm: 80 layers Γ 2 Γ 8192 Γ 2 bytes / 900 GB/s β 0.003 ms. Total: ~21 ms β. TP=4: Compute: 35 GB / 3.35 TB/s = 10.4 ms. Comm: 80 Γ 2 Γ 8192 Γ 2 / 450 GB/s β 0.006 ms. Total: ~10.4 ms β. TP=8: Compute: 17.5 GB / 3.35 TB/s = 5.2 ms. Comm: 80 Γ 2 Γ 8192 Γ 2 / 225 GB/s β 0.012 ms. Total: ~5.2 ms β. All meet 40ms, but TP=2 uses 2 GPUs ($5.60/hr) vs TP=8 using 8 GPUs ($22.40/hr). Optimal: TP=2 β it meets the SLA at 25% of the cost. TP=4 only if you need <15ms for streaming UX.
π Deep Dive: Training
</details>Interviewer: "You're building a continuous training pipeline that ingests 10 TB of new user interaction data daily. Last month, a silent data corruption (a logging schema change) poisoned 3 days of training data before anyone noticed, causing a 5% accuracy regression that took 2 weeks to diagnose. Design a data quality pipeline that catches this within 1 hour."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Add data validation checks before training." This is necessary but insufficient β schema validation catches structural changes but not statistical drift in valid-looking data.
Realistic Solution: A production data quality pipeline needs four layers: (1) Schema validation β Great Expectations or TFX Data Validation. Catches: missing columns, type changes, null rates. Latency: minutes. (2) Statistical profiling β compute per-column distributions (mean, std, quantiles, cardinality) on each hourly data shard and compare against a 7-day rolling baseline. Alert on KL-divergence > threshold. Catches: distribution shifts, logging bugs that produce valid-but-wrong values. (3) Embedding drift detection β run a small encoder on a sample of each shard, compare embedding centroids against baseline. Catches: semantic shifts that column-level stats miss. (4) Canary training β train a small proxy model on each day's data for 100 steps, compare loss curve against expected trajectory. Catches: data that's structurally and statistically valid but produces bad gradients.
Napkin Math: 10 TB/day = 417 GB/hour. Schema validation: trivial compute, <1 min. Statistical profiling on 1% sample: 4.17 GB Γ 100 columns Γ basic stats = ~5 min on 8 CPU cores. Embedding drift: encode 10k samples/hour with a small BERT β 30 sec on 1 GPU. Canary training: 100 steps on proxy model β 2 min on 1 GPU. Total detection latency: <10 minutes. Cost: 1 GPU + 8 CPU cores = ~$3/hour = $2,160/month. The schema change incident cost: 3 days of bad training ($15k in GPU waste) + 2 weeks of engineer time to diagnose (2 engineers Γ $100/hr Γ 80 hrs = $16k) + accuracy regression impact. Prevention ROI: $31k saved per incident vs $2.2k/month monitoring cost.
π Deep Dive: Data Engineering
</details>Interviewer: "Your team just completed a 30-day training run on 256 H100 GPUs. The cloud bill shows $650,000 in compute charges. Your sustainability team asks: 'What was the energy consumption and carbon footprint of this training run?' Calculate it for a US data center."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just multiply GPU TDP by hours." This ignores Power Usage Effectiveness (PUE) β the overhead of cooling, networking, storage, and facility infrastructure.
Realistic Solution: GPU power is only part of the picture. Data centers have a PUE (Power Usage Effectiveness) that captures total facility power divided by IT equipment power. A modern hyperscaler PUE is 1.1β1.2; an older facility might be 1.4β1.6. You must also account for the fact that GPUs don't always run at TDP β average utilization-weighted power is typically 60β80% of TDP.
Napkin Math: 256 H100s Γ 700W TDP = 179.2 kW IT load. Average utilization-weighted power: ~80% β 143 kW. PUE of 1.2: total facility power = 143 Γ 1.2 = 171.6 kW. Duration: 30 days Γ 24 hours = 720 hours. Energy consumed: 171.6 kW Γ 720 h = 123,552 kWh β 124 MWh. US average grid carbon intensity: 0.39 kg COβ/kWh. Carbon footprint: 124 MWh Γ 390 kg/MWh = 48.3 metric tons COβ. That's equivalent to ~10 passenger cars driven for a year. In a renewable-powered data center (0.05 kg/kWh): 6.2 tons COβ β an 87% reduction. Energy cost at $0.08/kWh: 124 MWh Γ $80/MWh = $9,900 β only 1.5% of the $650k cloud bill (the rest is margin, depreciation, and overhead).
π Deep Dive: Sustainable AI
</details>Interviewer: "You have a 70B FP16 model that costs $1.20 per 1M tokens to serve on 4Γ H100 GPUs. Your target is $0.40 per 1M tokens on a single H100. Design a compression pipeline that achieves this 3Γ cost reduction while keeping quality degradation under 2% on your evaluation suite."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just quantize to INT4 β that's 4Γ smaller, so it fits on one GPU." Naive INT4 quantization of a 70B model causes 5β10% quality loss, and fitting in memory doesn't mean meeting latency SLAs.
Realistic Solution: A production compression pipeline is multi-stage, not a single quantization step. The pipeline: (1) Calibration-aware quantization (GPTQ/AWQ) to INT4 with group size 128 β reduces weights from 140 GB to 35 GB. Quality loss: ~1β2% on most benchmarks. (2) KV-cache quantization to FP8 β halves KV-cache memory, enabling larger batch sizes. Quality impact: <0.5%. (3) Speculative decoding with a 1B draft model β the draft model proposes 4β5 tokens, the 70B model verifies in one forward pass. Increases effective throughput 2β3Γ for free. (4) Continuous batching (vLLM/TensorRT-LLM) β maximizes GPU utilization across concurrent requests.
Napkin Math: Original: 140 GB FP16 on 4Γ H100 (TP=4). After INT4: 35 GB β fits on 1Γ H100 (80 GB) with 45 GB for KV-cache. KV-cache in FP8 at 4k context: ~0.5 GB/request β batch of 80 concurrent requests. Speculative decoding: 2.5Γ effective token throughput. Continuous batching: 90% GPU utilization vs 40% with static batching. Net throughput per GPU: original system did X tokens/sec across 4 GPUs. Compressed system does ~1.2X tokens/sec on 1 GPU (INT4 2Γ compute + speculative 2.5Γ β overhead). At ~7.5M tokens/hr on a single $2.80/hr H100, the cost drops to $0.37 per 1M tokens vs the original $1.20. Quality: INT4 (β1.5%) + FP8 KV (β0.3%) = β1.8% total, under the 2% budget.
π Deep Dive: Model Compression
</details>Interviewer: "Draw the roofline model for an H100 GPU at four precisions: FP32, FP16/BF16, FP8, and INT8. For each precision, calculate the ridge point. Then explain why a workload that is compute-bound at FP16 can become memory-bound at INT8 β even though INT8 is 'faster.'"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Lower precision always makes things faster because you do more operations per second." This ignores that quantization changes both the compute ceiling AND the arithmetic intensity of the workload.
Realistic Solution: Each precision has a different peak FLOPS (compute ceiling) and the same memory bandwidth (3.35 TB/s on H100). The ridge point β where a workload transitions from memory-bound to compute-bound β shifts right as precision drops because compute scales faster than bandwidth. A workload with fixed arithmetic intensity can cross from above the ridge point (compute-bound) to below it (memory-bound) when you switch to a lower precision with a higher ridge point.
Napkin Math: H100 specs β FP32: 67 TFLOPS, BF16: 989 TFLOPS, FP8: 1,979 TFLOPS, INT8: 1,979 TOPS. Bandwidth: 3.35 TB/s (constant). Ridge points: FP32 = 67T / 3.35T = 20 Ops/Byte. BF16 = 989 / 3.35 = 295 Ops/Byte. FP8 = 1979 / 3.35 = 591 Ops/Byte. INT8 = 1979 / 3.35 = 591 Ops/Byte. Consider a large-batch GEMM with arithmetic intensity = 400 Ops/Byte. At BF16: 400 > 295 β compute-bound, attains 989 TFLOPS. At INT8: 400 < 591 β memory-bound, attains only 3.35T Γ 400 = 1,340 TOPS (not the 1,979 peak). The INT8 version is still faster (1,340 vs 989), but it's now leaving 32% of INT8 compute on the table. To fully utilize INT8, you need intensity > 591 β meaning larger batch sizes or fused operations.
π Deep Dive: Hardware Acceleration
</details>Interviewer: "You're training a 175B model on 2,048 H100 GPUs for 90 days. At this scale, the mean time between failures (MTBF) for any single GPU is ~1,000 hours, but with 2,048 GPUs, the cluster MTBF is under 30 minutes. Design a fault-tolerant training framework that achieves >95% effective utilization despite continuous hardware failures."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Checkpoint frequently and restart from the last checkpoint." At 30-minute MTBF, traditional checkpoint-restart spends more time recovering than training. Each restart requires: detect failure, drain pipeline, load checkpoint, re-warm data loaders, re-establish NCCL communicators β easily 10β20 minutes per event.
Realistic Solution: A fault-tolerant framework at this scale needs five mechanisms: (1) In-memory redundant checkpointing β replicate optimizer state across nodes so a failure doesn't require disk I/O. Each node stores its shard + one neighbor's shard in CPU RAM. Recovery: <30 seconds. (2) Elastic training β when a node fails, the surviving nodes redistribute work without stopping. Requires a topology-aware parallelism planner that can re-shard TP/PP/DP groups on the fly. (3) Hot spare pool β maintain 3β5% spare GPUs pre-loaded with the model. When a failure occurs, the spare swaps in while the failed node is replaced. (4) Hierarchical health monitoring β heartbeat every 5 seconds, with local (intra-node NVLink) and global (inter-node InfiniBand) failure domains. Detect failures in <10 seconds. (5) Asynchronous checkpointing β overlap checkpoint writes with training computation. Write to local NVMe first (fast), then async replicate to distributed storage.
Napkin Math: 2,048 GPUs, MTBF per GPU = 1,000 hrs. Cluster MTBF = 1,000 / 2,048 β 0.49 hrs β 29 minutes. Failures per 90-day run: 90 Γ 24 / 0.49 β 4,408 failures. Naive checkpoint-restart: 15 min recovery Γ 4,408 = 1,102 hours lost. Training time: 90 Γ 24 = 2,160 hrs. Effective utilization: (2,160 β 1,102) / 2,160 = 49%. With elastic training + hot spares: 30 sec recovery Γ 4,408 = 37 hours lost. Spare pool: 100 GPUs Γ 90 days = overhead. Effective utilization: (2,160 β 37) / 2,160 = 98.3%. Cost of spare pool: 100 GPUs Γ 2,160 hrs Γ $3.50 = $756k. Cost of 49% utilization loss: 1,102 hrs Γ 2,048 GPUs Γ $3.50 = $7.9M. Spare pool ROI: 10.4Γ.
π Deep Dive: Training
</details>Interviewer: "Your company has committed to carbon-neutral AI training by 2027. You run 500 training jobs per week across 3 data center regions (Virginia, Oregon, Ireland). Each region has different carbon intensity, electricity cost, and GPU availability. Design a carbon-aware job scheduler that minimizes carbon emissions while keeping total training cost within 15% of the carbon-unaware baseline."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just run everything in the greenest region." This ignores capacity constraints (the green region may not have enough GPUs), latency to data sources, and temporal variation in grid carbon intensity.
Realistic Solution: Carbon-aware scheduling is a multi-objective optimization problem with three dimensions: (1) Spatial shifting β route jobs to the region with the lowest current carbon intensity. (2) Temporal shifting β delay non-urgent jobs to hours when the grid is cleanest (solar peak, wind events). (3) Workload shaping β adjust GPU allocation to match renewable availability (scale up during clean hours, scale down during dirty hours). The scheduler needs a carbon intensity forecast API (e.g., WattTime, ElectricityMap) and a job priority system (urgent jobs run anywhere; batch jobs wait for clean windows).
Napkin Math: Regional carbon intensity: Virginia = 0.35 kg COβ/kWh (coal-heavy grid), Oregon = 0.08 kg/kWh (hydro), Ireland = 0.25 kg/kWh (mixed). Average job: 64 GPUs Γ 24 hrs Γ 700W Γ PUE 1.2 = 1,290 kWh. Carbon per job: Virginia = 451 kg, Oregon = 103 kg, Ireland = 323 kg. Carbon-unaware (uniform distribution): 500 jobs Γ (451+103+323)/3 = 146,200 kg COβ/week. Spatial-only (all to Oregon): 500 Γ 103 = 51,500 kg (β65%). But Oregon has capacity for only 200 jobs/week. Spatial + temporal: 200 jobs in Oregon, 150 in Ireland (shifted to wind hours, effective 0.15 kg/kWh), 150 in Virginia (shifted to solar hours, effective 0.20 kg/kWh). Carbon: 200Γ103 + 150Γ194 + 150Γ258 = 88,400 kg (β40%). Cost premium: Oregon GPUs cost 10% more, temporal shifting adds 8-hour average delay β 12% cost increase. Within the 15% budget, achieves 40% carbon reduction. Full carbon neutrality requires purchasing renewable energy certificates (RECs) for the remaining 88,400 kg at ~$15/ton = $1,326/week.
π Deep Dive: Sustainable AI
</details>Interviewer: "Your GPU cluster has grown organically over 3 years and now contains: 256Γ A100-40GB, 128Γ A100-80GB, 512Γ H100-80GB, and 64Γ H200-141GB. You run a mix of training jobs (10β1000 GPUs) and inference workloads. The current scheduler treats all GPUs as equivalent, leading to 35% average utilization. Design a heterogeneity-aware scheduler that maximizes cluster-wide utilization and cost efficiency."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Assign the newest GPUs to the most important jobs." This is priority scheduling, not heterogeneity-aware scheduling. It leaves old GPUs idle and doesn't account for workload-hardware affinity.
Realistic Solution: A heterogeneity-aware scheduler needs three components: (1) Workload profiling β automatically characterize each job's resource profile: compute-bound vs memory-bound (roofline), VRAM requirement, communication pattern (TP/DP/PP), and interconnect sensitivity. (2) Hardware-workload affinity scoring β match workloads to GPU types based on cost-efficiency, not raw performance. Memory-bound inference β H200 (best bandwidth/dollar). Compute-bound training β H100 (best FLOPS/dollar). Small fine-tuning β A100-40GB (cheapest, sufficient VRAM). Large-batch training β A100-80GB (good VRAM, acceptable FLOPS). (3) Bin-packing with fragmentation avoidance β pack small jobs onto partially-used nodes before allocating fresh nodes. Reserve contiguous NVLink domains for TP-heavy jobs.
Napkin Math: Current utilization: 35% across 960 GPUs. Effective GPU-hours/day: 960 Γ 24 Γ 0.35 = 8,064. Affinity-based scheduling: Route LLM inference (60% of workload, memory-bound) to H200s: 64 H200s at 85% util = 1,306 GPU-hrs. Route large training (25%) to H100s: 512 Γ 0.70 = 8,602 GPU-hrs. Route fine-tuning (15%) to A100s: 384 Γ 0.60 = 5,530 GPU-hrs. Total: 15,438 GPU-hrs/day β 91% increase in effective utilization. Cluster-wide utilization: ~67%. The remaining gap (67% β 85%) comes from bin-packing improvements and preemptible backfill jobs that run on idle GPUs. At $2.50/GPU-hr average, improving utilization from 35% to 70% saves: 960 Γ 24 Γ 0.35 Γ $2.50 = $20,160/day = $604k/month in recovered capacity.
π Deep Dive: Hardware Acceleration
</details>Interviewer: "Your platform team supports 200 ML teams deploying 1,000+ models per month. Each model currently requires manual optimization (quantization, graph compilation, kernel tuning) by a specialized ML systems engineer β a 2-week process per model. Design an automated model optimization pipeline that reduces this to <1 hour with <3% quality regression on 95% of models."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just run TensorRT on every model." TensorRT handles graph optimization and kernel selection, but doesn't handle quantization calibration, accuracy validation, or the heterogeneous model zoo (not all models are ONNX-exportable or have static shapes).
Realistic Solution: An automated optimization pipeline has five stages: (1) Model profiling β automatically characterize the model: architecture family (transformer, CNN, GNN), operator breakdown, memory footprint, arithmetic intensity. Takes ~5 minutes. (2) Optimization strategy selection β rule-based + learned policy. Transformers β quantize attention to FP8, MLP to INT8, keep embeddings in FP16. CNNs β INT8 throughout. GNNs β keep FP16 (sparse ops don't quantize well). (3) Calibration-aware quantization β run GPTQ/SmoothQuant with 512 calibration samples from the team's validation set. Takes ~20 minutes on 1 GPU. (4) Graph compilation β TensorRT/TVM/XLA with autotuning. Operator fusion, memory planning, kernel selection. Takes ~15 minutes. (5) Automated quality gate β run the team's evaluation suite, compare against FP16 baseline. If regression > 3%, fall back to a less aggressive quantization (e.g., INT8 β FP16 for sensitive layers identified by per-layer sensitivity analysis).
Napkin Math: Manual process: 2 weeks Γ 40 hrs Γ $100/hr = $8,000/model. 1,000 models/month = $8M/month in engineering time (impossible β you'd need 500 ML systems engineers). Reality: only 50 models/month get optimized; the rest run unoptimized (2β5Γ over-provisioned). Automated pipeline: 1 GPU-hour per model Γ $3.50 = $3.50/model. 1,000 models Γ $3.50 = $3,500/month. Success rate: 95% of models pass the 3% quality gate automatically. 5% (50 models) require human intervention β 50 Γ $8,000 = $400k. Total: $403k/month vs $8M theoretical (or vs massive over-provisioning). The real savings: 950 models now run optimized instead of 50. Average 2Γ throughput improvement β 50% GPU reduction across the fleet. At $5M/month GPU spend: $2.5M/month savings from automated optimization.
π Deep Dive: Model Compression
</details>Interviewer: "Your team has a critical recommendation model whose performance suddenly dropped by 15% AUC in production. The model was retrained just last week, and the new model performed excellently in staging. You suspect a data issue, but all data pipelines show 'success'. How do you quickly pinpoint if the input data for the production model is different from the training data, and what's the most common culprit?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Re-run the training job with production data" or "Look at the model's metrics in production." While useful, these are reactive. The core problem is lack of data versioning and immediate validation at the data ingress point.
Realistic Solution: The most common culprit is unversioned or silently changing upstream data sources. Even if the pipeline 'succeeded', the data it processed might have shifted. To quickly diagnose:
Napkin Math: If a 10TB dataset is split into 1000 partitions, hashing each partition can take time. A single
sha256sumon a 1GB file might take ~0.5 seconds on a typical cloud instance. For 1000 partitions, that's 500 seconds (~8 minutes). For 1000 partitions, comparing a few key statistics for each (mean, std dev, nulls) is much faster, perhaps 10ms per partition, totaling 10 seconds. Prioritize statistical checks for speed, then targeted hashing if statistics diverge.
Key Equation: $H(D_{prod}) \stackrel{?}{=} H(D_{train})$ where $H$ is a cryptographic hash function and $D$ is the dataset.
π Deep Dive: Volume I: Data Versioning
</details>Interviewer: "Your company operates a critical real-time fraud detection model. Features are sourced from an online feature store (e.g., Redis, DynamoDB) and an offline batch feature store (e.g., Hive, BigQuery). Recently, you've observed that the online model's performance degrades significantly shortly after a new batch of features is pushed to the offline store, even though the model itself hasn't changed. Describe the likely architectural flaw and propose a solution that ensures consistency and freshness without introducing excessive latency."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "The online model is buggy" or "The offline features are bad." The issue is not necessarily bad data or a bad model, but an inconsistency in the timing or method of feature updates between online and offline stores, leading to a train-serve skew.
Realistic Solution: The likely flaw is a train-serve skew due to asynchronous or inconsistent updates between the online and offline feature stores. When new features are generated offline (e.g., daily aggregations), they are used to retrain the model. However, if these new features are not immediately or consistently propagated to the online store, the online model will be served with stale features while being trained on fresh ones. This creates a mismatch.
A robust solution involves ensuring the online and offline feature stores are synchronized and consistent:
Napkin Math: If an offline feature computation takes 4 hours and runs daily, and the online store is updated only after the model is retrained and deployed (which might take another 2 hours), there's a potential 6-hour window of inconsistency. If an online feature update takes 50ms to propagate to Redis and 200ms to Kafka, total latency for dual-write is dominated by the slower path, but for consistency, both must succeed. Aim for sub-second synchronization for critical features.
Key Equation: $F_{online}(t) \approx F_{offline}(t - \Delta t_{sync})$ where $\Delta t_{sync}$ should be minimized.
π Deep Dive: Volume I: Feature Stores
</details>Interviewer: "You oversee a critical ML pipeline processing petabytes of data daily, feeding hundreds of downstream models across your organization. Last quarter, an upstream team made a seemingly minor schema change in their raw data source (e.g., changed an integer column to a float, or made a non-nullable column nullable). This change wasn't communicated, and it silently propagated through your pipeline, causing subtle but widespread model degradation and hard-to-debug issues for weeks before detection. Design a robust, scalable system to prevent such 'silent schema shifts' from impacting downstream ML models."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just add more unit tests to the data processing code." While good, unit tests often only catch issues within the transformation logic, not changes to the source data's inherent properties. Relying on manual communication is also error-prone.
Realistic Solution: The core problem is a lack of data contracts and automated schema validation with lineage awareness. A robust system would involve:
INT to FLOAT), define clear, automated coercion rules within the pipeline to prevent downstream errors, while still logging the change.Napkin Math: Validating the schema of a 1TB Parquet file might involve reading its metadata (header) which is fast (milliseconds), and then sampling data to check actual values against data quality rules. If a sample of 10,000 rows (e.g., 10MB) is validated, this takes seconds. For petabytes of data, this scales by processing partitions independently. On a cluster with 100 nodes, validating 100,000 partitions (100GB each) could be done in minutes.
Key Equation: $S_{actual}(D) \stackrel{?}{=} S_{contract}(D)$ where $S$ is the schema and $D$ is the dataset.
π Deep Dive: Volume I: Data Quality & Contracts
</details>Interviewer: "Your organization's data lake, primarily on S3, has grown to 500 PB of raw and processed data. The monthly storage bill is astronomical and constantly increasing. Much of this data is historical, accessed infrequently (e.g., for compliance audits, long-tail research, or rare model re-trainings). As a Principal ML Systems Engineer, you're tasked with drastically reducing this cost without compromising data availability for critical ML workloads or compliance. Detail your strategy, including specific AWS services and architectural considerations."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just delete old data" or "Move everything to Glacier." Deleting data can violate compliance or hinder future research. Moving everything to Glacier without careful analysis can lead to high retrieval costs and latency for necessary access.
Realistic Solution: The strategy revolves around intelligent data tiering, lifecycle management, and optimizing storage formats, recognizing that not all data has the same access patterns or value over time.
access_frequency (hot, warm, cold), retention_policy, data_domain, owner.Napkin Math:
- S3 Standard: ~$0.023/GB/month
- S3 Standard-IA: ~$0.0125/GB/month (plus retrieval fees)
- S3 Glacier Deep Archive: ~$0.00099/GB/month (plus retrieval fees) If 70% of 500 PB (350 PB) can move from S3 Standard to Deep Archive: Current cost (Standard): $0.023 * 500 PB = $11.5M/month Optimized cost: (0.023 * 150 PB) + (0.00099 * 350 PB) = $3.45M + $0.3465M = ~$3.8M/month. This is a potential savings of ~$7.7M/month. Retrieval costs for Glacier Deep Archive can be high (e.g., $0.0025/GB for bulk retrieval, taking 12+ hours). Retrieving 100TB would cost $250, but take significant time.
Key Equation: $TotalCost = \sum_{i=1}^{N} (StorageCost_i \times DataVolume_i) + \sum_{j=1}^{M} (RetrievalCost_j \times DataRetrieved_j)$
π Deep Dive: Volume I: Cloud Storage & Cost Optimization
</details>Interviewer: "Your team is training a large vision model on 32 A100 GPUs distributed across 8 instances, using PyTorch DDP. You notice that while GPU utilization is high during the initial epochs, it frequently drops to 40-50% during later epochs, despite sufficient CPU and memory on each instance. nvidia-smi shows GPUs are often idle for short bursts. The data is stored in a shared S3 bucket and loaded via torch.utils.data.DataLoader with multiple worker processes. What is the most likely bottleneck, and how would you systematically diagnose and resolve it?"
Common Mistake: "Increase batch size" or "Optimize model architecture." While these can help GPU utilization, the observation of drops later in training and short idle bursts points away from model-internal issues and towards data I/O.
Realistic Solution: The most likely bottleneck is data loading and I/O from S3. As training progresses, the data access patterns might become less cache-friendly, or network contention for S3 access becomes an issue, especially with multiple workers across many instances simultaneously fetching data. The short bursts of GPU idle time indicate the GPUs are "waiting" for data from the CPU/I/O subsystem.
Diagnosis:
torch.profiler) or custom timers around the next(iter(dataloader)) call to measure the time spent loading a batch. Compare this against the time spent on forward/backward passes.DataLoader worker processes. Look for spikes in network latency or high CPU usage in data loading workers.Resolution:
rsync scripts) to pre-fetch relevant shards to each node.num_workers: Experiment with num_workers in DataLoader. Too few workers can underutilize CPU and I/O; too many can lead to contention or excessive memory usage.pin_memory=True: For PyTorch, setting pin_memory=True can speed up data transfer from CPU to GPU.DataLoader's prefetch_factor to ensure the next batch is ready before the current one finishes processing.Napkin Math: A single A100 GPU can process ~1000 images/sec (batch size 128, typical ResNet). With 32 GPUs, this is 32,000 images/sec. If each image is 1MB, total data throughput needed is 32 GB/sec. A single
c5n.18xlargeinstance has up to 100 Gbps (12.5 GB/s) network bandwidth. 8 instances could theoretically provide 100 GB/s. However, S3 throughput per object can be limited, and network bottlenecks often arise from shared paths or single-threaded data loaders. Local NVMe SSDs can deliver 5-10 GB/s per instance. Caching 1TB of data on local SSDs would take ~2 minutes at 8 GB/s.
Key Equation: $T_{total} = T_{data_load} + T_{gpu_compute} + T_{comm}$ (minimize $T_{data_load}$)
π Deep Dive: Volume I: Data Loading for Distributed Training
</details>Interviewer: "Your ML team maintains a critical production model that predicts user engagement. For the past two months, the model's business impact has been steadily declining, but no alarms have fired from your standard model monitoring dashboards (e.g., accuracy, precision, recall). What are the common reasons for this 'silent decay,' and what additional monitoring would you implement to detect it proactively?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "The model needs retraining." While often true, the question focuses on detection when standard metrics don't alert. The problem is that standard metrics often compare predictions to ground truth, which might be delayed or unavailable, or they don't capture shifts in the input data or underlying relationships.
Realistic Solution: The 'silent decay' is typically due to data drift or concept drift that isn't immediately reflected in traditional performance metrics.
user_device_type suddenly sees a spike in a new device category not present during training.Additional Monitoring to Implement:
Napkin Math: If monitoring 100 features, and each statistical test takes 50ms, then running checks for all features takes 5 seconds per hour. Over a month, this is 5s * 24 * 30 = 3600 seconds = 1 hour of compute. This is negligible compared to the cost of model decay.
Key Equation: $D(P_{prod}(X) || P_{train}(X))$ where $D$ is a statistical divergence metric (e.g., KL-divergence, JS-divergence, KS-statistic) and $P(X)$ is the probability distribution of input features.
π Deep Dive: Volume I: Model Monitoring & Drift Detection
</details>Interviewer: "Your company operates in a highly regulated industry (e.g., healthcare, finance) and needs to train a powerful new ML model using customer data that contains sensitive Personally Identifiable Information (PII). Direct use of this PII for training is prohibited due to privacy regulations (GDPR, CCPA, HIPAA) and corporate policy. Propose and compare at least three distinct, highly technical approaches to enable model training while strictly preserving user privacy and complying with regulations. Discuss their trade-offs in terms of model utility, implementation complexity, and computational cost."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just anonymize the data." Simple anonymization (e.g., hashing, pseudonymization) is often insufficient as re-identification attacks are sophisticated, and it can significantly reduce model utility.
Realistic Solution: This problem requires advanced privacy-preserving ML (PPML) techniques.
Other Considerations (less distinct but complementary):
Napkin Math:
- FL: Training a model on 100M users, each with a small local dataset, could involve 100M gradient updates. If each update is 1MB, total communication is 100TB per epoch. With DP, each client might add small noise.
- Synthetic Data: Training a GAN on a 1TB PII dataset might require 100s of GPU hours (e.g., 50 A100-hours).
- DP: Adding noise to gradients in a typical neural network might increase training time by 5-10% due to specialized optimizers (e.g., Opacus for PyTorch).
Key Equation: $\epsilon$-Differential Privacy: $\forall x, x' \text{ differing on one individual, and } \forall S \subseteq Range(A)$, $\Pr[A(x) \in S] \le e^\epsilon \Pr[A(x') \in S] + \delta$.
π Deep Dive: Volume I: Privacy-Preserving ML
</details>Interviewer: "A critical model deployed six months ago is failing in production, and your team needs to quickly debug it. However, you discover that you cannot reliably reproduce the exact training run: the model's performance on the original test set differs significantly, and you can't trace back the specific data, code, or environment that produced the deployed model. As a Principal Engineer, outline a comprehensive MLOps strategy to ensure full reproducibility for all future models, considering large datasets, complex dependencies, and distributed training environments."
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just commit the code to Git." Code is only one piece of the puzzle. Data, dependencies, environment, and hyperparameters are equally crucial.
Realistic Solution: Full ML reproducibility requires a holistic approach to versioning and tracking all artifacts involved in the ML lifecycle.
pip freeze > requirements.txt or conda env export > environment.yml to lock exact dependency versions.Napkin Math: Storing 1000 models, each with 100MB weights and 10MB of associated artifacts (logs, configs), is 110GB. This is trivial storage. The complexity is linking these artifacts correctly. A single training run might generate 100s of metrics, 50 hyperparameters, and multiple artifacts. A well-designed experiment tracking system handles this overhead.
Key Equation: $Model_{reproducible} = F(Code_{version}, Data_{version}, Env_{version}, HParams_{version})$
π Deep Dive: Volume I: MLOps & Reproducibility
</details>Interviewer: "Your company's flagship recommendation model is trained by a complex, multi-stage ML pipeline orchestrated by Airflow/Kubeflow. The pipeline often takes 12+ hours to complete. Recently, you've observed frequent intermittent failures in one of the intermediate stages (e.g., a transient network error, or a temporary resource exhaustion). When this happens, the entire pipeline restarts from the very beginning, wasting significant compute resources and delaying model updates. How would you redesign this pipeline to be more fault-tolerant and cost-efficient, specifically focusing on making its stages idempotent?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "Just increase resource limits." While this might reduce failures, it doesn't address the fundamental issue of wasteful restarts and doesn't make the pipeline robust to any failure.
Realistic Solution: The core problem is a lack of idempotency and effective checkpointing within the pipeline stages. An idempotent operation is one that can be applied multiple times without changing the result beyond the initial application.
Redesign Strategy:
Napkin Math: If a 12-hour pipeline has 4 stages of 3 hours each, and a failure in stage 3 causes a full restart, you lose 6 hours of compute. With idempotency, if stage 1 and 2 completed successfully and their outputs are valid, a restart only re-runs stage 3 (and possibly stage 4), saving 6 hours. Over 10 failures a month, that's 60 hours saved. If the compute costs $100/hour, that's $6000/month in direct savings, plus faster model updates.
Key Equation: $Cost_{saved} = N_{failures} \times (T_{total_pipeline} - T_{remaining_stages})$
π Deep Dive: Volume I: MLOps Pipelines & Idempotency
</details>Interviewer: "You have a PyTorch training pipeline. You set torch.manual_seed(42), np.random.seed(42), and random.seed(42). You train a model twice on the exact same dataset on the exact same machine. The loss curves diverge after the first epoch. What standard PyTorch DataLoader argument completely destroys your random seed determinism, and why?"
Common Mistake: "GPU floating-point math is non-deterministic." While GPU atomics can introduce slight variations, complete loss curve divergence after epoch 1 points directly to the data loading order.
Realistic Solution: The culprit is num_workers > 0 combined with shuffle=True.
When you set num_workers=4, PyTorch forks 4 separate background processes to load data. The OS process scheduler determines exactly when each worker wakes up, reads a file from disk, and pushes it into the shared memory queue.
Even if each worker is seeded perfectly and shuffles its own chunk of data deterministically, the interleaving of batches arriving from the 4 workers into the main training thread is entirely dependent on OS-level thread scheduling jitter and disk I/O latency. Batch A might arrive before Batch B on Tuesday, but Batch B beats Batch A on Wednesday. The model sees the data in a different order, destroying reproducibility.
The Fix: You must use a worker_init_fn to correctly seed each worker based on its ID and the current epoch, and if absolute bit-for-bit determinism is required, you must sort/sequence the outputs of the workers before feeding them to the GPU (which limits throughput), or simply accept the stochasticity of multi-process data loading.
Napkin Math: In an epoch of 1,000,000 images, 4 workers are racing to deliver 31,250 batches of 32. The combinatorial explosion of possible delivery interleavings guarantees the GPU will never see the exact same sequence of batches twice across different runs.
π Deep Dive: Volume I: Data Engineering
</details>Interviewer: "Your infrastructure team requires saving a checkpoint every 30 minutes to an AWS S3 bucket. Your 70B model checkpoint is 140 GB. When the checkpoint function is called, GPU utilization drops to 0% for almost 3 minutes. Your team suggests upgrading to a faster S3 tier. Why won't faster S3 fix the 3-minute stall?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "S3 upload bandwidth is the bottleneck." S3 can easily handle massive parallel throughput. The bottleneck is the CPU serialization process.
Realistic Solution: The stall is caused by the Pickle/Safetensors Serialization wall on the CPU.
When you call torch.save(), PyTorch must take 140 GB of tensors sitting in GPU HBM and:
pickle, or ideally safetensors).Python's pickle is heavily CPU-bound and often single-threaded. Serializing 140 GB of data on a single CPU core is agonizingly slow, regardless of how fast your network pipe to S3 is. The GPUs sit completely idle while a single CPU core chugs through gigabytes of Python object serialization.
The Fix:
memcpy the tensors to host RAM, and then a background thread handles the slow serialization and S3 upload, allowing the GPUs to resume computing the next batch instantly.pickle to safetensors, which enables zero-copy memory mapping and bypasses Python's serialization overhead entirely.Napkin Math: Single-threaded
pickleserialization speed: ~800 MB/s. 140,000 MB / 800 MB/s = 175 seconds (nearly 3 minutes) of pure CPU bottleneck before the data even hits the network interface.
π Deep Dive: Volume II: Fault Tolerance
</details>Interviewer: "Your company has 10 Petabytes of multimodal training data sitting in AWS S3 in us-east-1. Due to a massive GPU shortage, you secured 2,000 H100s in a specialized cloud provider (like CoreWeave or Lambda Labs) located in Texas. How do you engineer the training pipeline to connect the data to the GPUs, and what is the hidden economic catastrophe?"
Common Mistake: "Just stream the data directly from S3 to the GPUs in Texas using PyTorch IterableDataset."
Realistic Solution: You fell victim to Data Gravity and Egress Fees.
Streaming 10 PB of data across the public internet has two fatal flaws:
The Fix: You cannot stream. You must physically or logically migrate the data once to a high-performance parallel file system (like Lustre or Weka) sitting physically adjacent to the GPUs in the Texas datacenter before training begins.
Napkin Math: 10 Petabytes = 10,000,000 Gigabytes. At $0.05 per GB, AWS will charge you $500,000 in a single egress fee just to read your dataset once. If your data loader streams multiple epochs without local caching, you pay that $500,000 penalty every single epoch.
π Deep Dive: Volume I: Data Engineering
</details>Interviewer: "Your model weights are loaded and served in FP16 (2 bytes per parameter). The model is 70 billion parameters, so you allocate 140 GB of disk space for the checkpoint file. But the training team sends over a .pt file that is 420 GB. What is taking up the extra 280 GB, and can you delete it before deploying?"
Common Mistake: "The file contains the training data or gradients." Training data isn't saved in model checkpoints, and gradients are ephemeral.
Realistic Solution: The extra space is consumed by the Optimizer States.
When training a model with an optimizer like Adam, the system must maintain moving averages of the gradients (momentum and variance) for every single parameter. Furthermore, for numerical stability, these statesβand often a master copy of the weights themselvesβare typically stored in FP32 (4 bytes per parameter).
70B Parameters * 4 bytes (Momentum) = 280 GB. 70B Parameters * 4 bytes (Variance) = 280 GB.
A full training checkpoint for a 70B model is actually pushing 700+ GB. The 420 GB file means they likely discarded the FP32 master weights but kept the FP16 weights + FP32 optimizer states (140GB + 280GB).
The Fix: Yes, for serving/deployment, the optimizer states are entirely useless. You should extract only the FP16 model.state_dict() and save it to a clean 140 GB file, discarding the optimizer states to save storage and drastically speed up model load times.
Napkin Math: 1 parameter = 2 bytes (FP16 weight) + 4 bytes (Adam m) + 4 bytes (Adam v). The optimizer state is exactly 4x larger than the FP16 inference weights.
π Deep Dive: Volume I: ML Frameworks
</details>Interviewer: "Your 512-GPU training job occasionally stalls for exactly 60 seconds. The hardware is healthy, no preemptions occurred, and the network is uncongested. You eventually trace the stall to a cron job running on the Linux host OS of a single node. What is the cron job doing that halts the entire cluster?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "The cron job is using all the CPU cores, starving the data loader." CPU starvation causes a slowdown, but a strict 60-second complete halt is a synchronization deadlock.
Realistic Solution: The cron job is performing Log Rotation (logrotate).
When logrotate runs, it typically copies the active log file, compresses it, and then sends a SIGHUP or restarts the daemon that is writing the logs. If your ML training script is writing massive amounts of telemetry to stdout (e.g., loss values every step) and piping it to a file, the logrotate operation can momentarily block the file descriptor.
If that single node blocks on I/O for just a few seconds, it falls behind. Because the 512 GPUs are running synchronous Data Parallelism (using AllReduce), the other 511 GPUs finish their math and hit the synchronization barrier. They wait. They wait until the straggler node finally finishes its I/O block, catches up on the math, and joins the collective communication ring. One blocked file descriptor on one node stalls all 512 GPUs.
The Fix:
Napkin Math: 511 GPUs sitting idle for 60 seconds = 30,660 GPU-seconds wasted. At $3.00/hr per GPU, that cron job just burned $25 of cloud credits in a single minute, entirely destroying your MFU.
π Deep Dive: Volume I: ML Operations
</details>Interviewer: "Your data engineering team provides a 5 TB dataset stored in Parquet format in AWS S3. Your PyTorch dataloader only needs to read a single column (the text feature) to train an NLP model. Because Parquet is columnar, this should be incredibly fast and save bandwidth. However, your dataloader is downloading data at 10 GB/s, saturating the network, and reading almost the entire 5 TB. Why didn't the columnar format save you bandwidth?"
<details> <summary><b>π Reveal Answer</b></summary>Common Mistake: "S3 doesn't support columnar reads." S3 supports byte-range requests perfectly. The issue is how the Parquet file was internally structured.
Realistic Solution: The dataset was written with massive Row Groups.
While Parquet is a columnar format, it actually partitions data horizontally into "Row Groups" first, and then stores the columns sequentially within those row groups.
If the data engineering team wrote the file using a single, massive Row Group (or very few, large ones), the physical layout on disk looks like this: [Col1_Chunk1, Col2_Chunk1, ..., ColN_Chunk1].
To read just Column 2, your dataloader must issue an HTTP Byte-Range request to S3. If the Row Group is massive, the byte range for Col2_Chunk1 might be gigabytes long. Worse, many naive dataloaders or Parquet readers will simply download the entire Row Group into memory just to extract the single column they need.
The Fix: You must optimize the Parquet writing process. Write the data using smaller, highly tuned Row Group sizes (e.g., 100 MB). This allows the PyTorch dataloader to issue highly precise, small byte-range requests to S3, fetching exactly the column chunks it needs and skipping the 90% of the file containing the unused columns.
Napkin Math: 5 TB file with 10 columns. If correctly chunked, reading 1 column pulls ~500 GB over the network. If written as a single Row Group, naive readers will download the entire 5,000 GB, wasting 4.5 TB of AWS egress and bandwidth.
π Deep Dive: Volume I: Data Engineering
</details>