````{grid} 1 1 3 3
:gutter: 3

```{grid-item-card} 🎧 Audio Overview

Listen to an AI-generated overview.

<audio controls style="width: 100%; height: 54px; margin-top: auto;">
  <source src="https://github.com/harvard-edge/cs249r_book/releases/download/tinytorch-audio-v0.1.1/19_benchmarking.mp3" type="audio/mpeg">
</audio>
```

```{grid-item-card} 🚀 Launch Binder

Run interactively in your browser.

<a href="https://mybinder.org/v2/gh/harvard-edge/cs249r_book/main?labpath=tinytorch%2Fmodules%2F19_benchmarking%2Fbenchmarking.ipynb" target="_blank" style="display: flex; align-items: center; justify-content: center; width: 100%; height: 54px; margin-top: auto; background: #f97316; color: white; text-align: center; text-decoration: none; border-radius: 27px; font-size: 14px; box-sizing: border-box;">Open in Binder →</a>
```

```{grid-item-card} 📄 View Source

Browse the source code on GitHub.

<a href="https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/19_benchmarking/19_benchmarking.py" target="_blank" style="display: flex; align-items: center; justify-content: center; width: 100%; height: 54px; margin-top: auto; background: #14b8a6; color: white; text-align: center; text-decoration: none; border-radius: 27px; font-size: 14px; box-sizing: border-box;">View on GitHub →</a>
```

````

```{raw} html
<style>
.slide-viewer-container {
  margin: 0.5rem 0 1.5rem 0;
  background: #0f172a;
  border-radius: 1rem;
  overflow: hidden;
  box-shadow: 0 4px 20px rgba(0,0,0,0.15);
}
.slide-header {
  display: flex;
  align-items: center;
  justify-content: space-between;
  padding: 0.6rem 1rem;
  background: rgba(255,255,255,0.03);
}
.slide-title {
  display: flex;
  align-items: center;
  gap: 0.5rem;
  color: #94a3b8;
  font-weight: 500;
  font-size: 0.85rem;
}
.slide-subtitle {
  color: #64748b;
  font-weight: 400;
  font-size: 0.75rem;
}
.slide-toolbar {
  display: flex;
  align-items: center;
  gap: 0.375rem;
}
.slide-toolbar button {
  background: transparent;
  border: none;
  color: #64748b;
  width: 32px;
  height: 32px;
  border-radius: 0.375rem;
  cursor: pointer;
  font-size: 1.1rem;
  transition: all 0.15s;
  display: flex;
  align-items: center;
  justify-content: center;
}
.slide-toolbar button:hover {
  background: rgba(249, 115, 22, 0.15);
  color: #f97316;
}
.slide-nav-group {
  display: flex;
  align-items: center;
}
.slide-page-info {
  color: #64748b;
  font-size: 0.75rem;
  padding: 0 0.5rem;
  font-weight: 500;
}
.slide-zoom-group {
  display: flex;
  align-items: center;
  margin-left: 0.25rem;
  padding-left: 0.5rem;
  border-left: 1px solid rgba(255,255,255,0.1);
}
.slide-canvas-wrapper {
  display: flex;
  justify-content: center;
  align-items: center;
  padding: 0.5rem 1rem 1rem 1rem;
  min-height: 380px;
  background: #0f172a;
}
.slide-canvas {
  max-width: 100%;
  max-height: 350px;
  height: auto;
  border-radius: 0.5rem;
  box-shadow: 0 4px 24px rgba(0,0,0,0.4);
}
.slide-progress-wrapper {
  padding: 0 1rem 0.5rem 1rem;
}
.slide-progress-bar {
  height: 3px;
  background: rgba(255,255,255,0.08);
  border-radius: 1.5px;
  overflow: hidden;
  cursor: pointer;
}
.slide-progress-fill {
  height: 100%;
  background: #f97316;
  border-radius: 1.5px;
  transition: width 0.2s ease;
}
.slide-loading {
  color: #f97316;
  font-size: 0.9rem;
  display: flex;
  align-items: center;
  gap: 0.5rem;
}
.slide-loading::before {
  content: '';
  width: 18px;
  height: 18px;
  border: 2px solid rgba(249, 115, 22, 0.2);
  border-top-color: #f97316;
  border-radius: 50%;
  animation: slide-spin 0.8s linear infinite;
}
@keyframes slide-spin {
  to { transform: rotate(360deg); }
}
.slide-footer {
  display: flex;
  justify-content: center;
  gap: 0.5rem;
  padding: 0.6rem 1rem;
  background: rgba(255,255,255,0.02);
  border-top: 1px solid rgba(255,255,255,0.05);
}
.slide-footer a {
  display: inline-flex;
  align-items: center;
  gap: 0.375rem;
  background: #f97316;
  color: white;
  padding: 0.4rem 0.9rem;
  border-radius: 2rem;
  text-decoration: none;
  font-weight: 500;
  font-size: 0.75rem;
  transition: all 0.15s;
}
.slide-footer a:hover {
  background: #ea580c;
  color: white;
}
.slide-footer a.secondary {
  background: transparent;
  color: #94a3b8;
  border: 1px solid rgba(255,255,255,0.15);
}
.slide-footer a.secondary:hover {
  background: rgba(255,255,255,0.05);
  color: #f8fafc;
}
@media (max-width: 600px) {
  .slide-header { flex-direction: column; gap: 0.5rem; padding: 0.5rem 0.75rem; }
  .slide-toolbar button { width: 28px; height: 28px; }
  .slide-canvas-wrapper { min-height: 260px; padding: 0.5rem; }
  .slide-canvas { max-height: 220px; }
}
</style>

<div class="slide-viewer-container" id="slide-viewer-19_benchmarking">
  <div class="slide-header">
    <div class="slide-title">
      <span>🔥</span>
      <span>Slide Deck</span>
      <span class="slide-subtitle">· AI-generated</span>
    </div>
    <div class="slide-toolbar">
      <div class="slide-nav-group">
        <button onclick="slideNav('19_benchmarking', -1)" title="Previous">‹</button>
        <span class="slide-page-info"><span id="slide-num-19_benchmarking">1</span> / <span id="slide-count-19_benchmarking">-</span></span>
        <button onclick="slideNav('19_benchmarking', 1)" title="Next">›</button>
      </div>
      <div class="slide-zoom-group">
        <button onclick="slideZoom('19_benchmarking', -0.25)" title="Zoom out">−</button>
        <button onclick="slideZoom('19_benchmarking', 0.25)" title="Zoom in">+</button>
      </div>
    </div>
  </div>
  <div class="slide-canvas-wrapper">
    <div id="slide-loading-19_benchmarking" class="slide-loading">Loading slides...</div>
    <canvas id="slide-canvas-19_benchmarking" class="slide-canvas" style="display:none;"></canvas>
  </div>
  <div class="slide-progress-wrapper">
    <div class="slide-progress-bar" onclick="slideProgress('19_benchmarking', event)">
      <div class="slide-progress-fill" id="slide-progress-19_benchmarking" style="width: 0%;"></div>
    </div>
  </div>
  <div class="slide-footer">
    <a href="../_static/slides/19_benchmarking.pdf" download>⬇ Download</a>
    <a href="#" onclick="slideFullscreen('19_benchmarking'); return false;" class="secondary">⛶ Fullscreen</a>
  </div>
</div>

<script src="https://cdnjs.cloudflare.com/ajax/libs/pdf.js/3.11.174/pdf.min.js"></script>
<script>
(function() {
  if (window.slideViewersInitialized) return;
  window.slideViewersInitialized = true;

  pdfjsLib.GlobalWorkerOptions.workerSrc = 'https://cdnjs.cloudflare.com/ajax/libs/pdf.js/3.11.174/pdf.worker.min.js';

  window.slideViewers = {};

  window.initSlideViewer = function(id, pdfUrl) {
    const viewer = { pdf: null, page: 1, scale: 1.3, rendering: false, pending: null };
    window.slideViewers[id] = viewer;

    const canvas = document.getElementById('slide-canvas-' + id);
    const ctx = canvas.getContext('2d');

    function render(num) {
      viewer.rendering = true;
      viewer.pdf.getPage(num).then(function(page) {
        const viewport = page.getViewport({scale: viewer.scale});
        canvas.height = viewport.height;
        canvas.width = viewport.width;
        page.render({canvasContext: ctx, viewport: viewport}).promise.then(function() {
          viewer.rendering = false;
          if (viewer.pending !== null) { render(viewer.pending); viewer.pending = null; }
        });
      });
      document.getElementById('slide-num-' + id).textContent = num;
      document.getElementById('slide-progress-' + id).style.width = (num / viewer.pdf.numPages * 100) + '%';
    }

    function queue(num) { if (viewer.rendering) viewer.pending = num; else render(num); }

    pdfjsLib.getDocument(pdfUrl).promise.then(function(pdf) {
      viewer.pdf = pdf;
      document.getElementById('slide-count-' + id).textContent = pdf.numPages;
      document.getElementById('slide-loading-' + id).style.display = 'none';
      canvas.style.display = 'block';
      render(1);
    }).catch(function() {
      document.getElementById('slide-loading-' + id).innerHTML = 'Unable to load. <a href="' + pdfUrl + '" style="color:#f97316;">Download PDF</a>';
    });

    viewer.queue = queue;
  };

  window.slideNav = function(id, dir) {
    const v = window.slideViewers[id];
    if (!v || !v.pdf) return;
    const newPage = v.page + dir;
    if (newPage >= 1 && newPage <= v.pdf.numPages) { v.page = newPage; v.queue(newPage); }
  };

  window.slideZoom = function(id, delta) {
    const v = window.slideViewers[id];
    if (!v) return;
    v.scale = Math.max(0.5, Math.min(3, v.scale + delta));
    v.queue(v.page);
  };

  window.slideProgress = function(id, event) {
    const v = window.slideViewers[id];
    if (!v || !v.pdf) return;
    const bar = event.currentTarget;
    const pct = (event.clientX - bar.getBoundingClientRect().left) / bar.offsetWidth;
    const newPage = Math.max(1, Math.min(v.pdf.numPages, Math.ceil(pct * v.pdf.numPages)));
    if (newPage !== v.page) { v.page = newPage; v.queue(newPage); }
  };

  window.slideFullscreen = function(id) {
    const el = document.getElementById('slide-viewer-' + id);
    if (el.requestFullscreen) el.requestFullscreen();
    else if (el.webkitRequestFullscreen) el.webkitRequestFullscreen();
  };
})();

initSlideViewer('19_benchmarking', '../_static/slides/19_benchmarking.pdf');
</script>
```

- **Implement** professional benchmarking infrastructure with statistical analysis including confidence intervals and variance control
- **Master** multi-objective optimization trade-offs between accuracy, latency, and memory through Pareto frontier analysis
- **Understand** measurement uncertainty, warmup protocols, and reproducibility requirements for fair model comparison
- **Connect** optimization techniques from Modules 14-18 into systematic evaluation workflows for the TorchPerf Olympics capstone

:align: center
:caption: Benchmarking Infrastructure
flowchart TB
    subgraph "Benchmarking Infrastructure"
        A["BenchmarkResult
Statistical container"]
        B["Benchmark
Single metric runner"]
        C["BenchmarkSuite
Multi-metric evaluation"]
        D["TorchPerf Olympics
Competition framework"]
    end

    E["Models to Compare"] --> B
    F["Test Datasets"] --> B
    B --> A
    A --> C
    C --> D

    style A fill:#e1f5ff
    style B fill:#fff3cd
    style C fill:#f8d7da
    style D fill:#d4edda

# Compare baseline vs optimized model with statistical rigor
benchmark = Benchmark([baseline_model, optimized_model])
latency_results = benchmark.run_latency_benchmark()
# Output: baseline: 12.3ms ± 0.8ms, optimized: 4.1ms ± 0.3ms (67% reduction, p < 0.01)

BenchmarkResult(metric_name: str, values: List[float], metadata: Dict[str, Any] = {})

with precise_timer() as timer:
    # Your code to measure
    ...
# Access elapsed time
elapsed_seconds = timer.elapsed

Benchmark(models: List[Any], datasets: List[Any],
          warmup_runs: int = 5, measurement_runs: int = 10)

BenchmarkSuite(models: List[Any], datasets: List[Any], output_dir: str = "benchmark_results")

# Warmup protocol in action
for _ in range(warmup_runs):
    _ = model(input_data)  # Run but discard measurements

def run_latency_benchmark(self, input_shape: Tuple[int, ...] = (1, 28, 28)) -> Dict[str, BenchmarkResult]:
    """Benchmark model inference latency using Profiler."""
    results = {}

    for i, model in enumerate(self.models):
        model_name = getattr(model, 'name', f'model_{i}')
        latencies = []

        # Create input tensor for this benchmark
        input_data = Tensor(np.random.randn(*input_shape).astype(np.float32))

        # Warmup runs (discard results)
        for _ in range(self.warmup_runs):
            if hasattr(model, 'forward'):
                _ = model.forward(input_data)
            elif callable(model):
                _ = model(input_data)

        # Measurement runs (collect statistics)
        for _ in range(self.measurement_runs):
            with precise_timer() as timer:
                if hasattr(model, 'forward'):
                    _ = model.forward(input_data)
                elif callable(model):
                    _ = model(input_data)
            latencies.append(timer.elapsed)

        results[model_name] = BenchmarkResult(
            metric_name=f"{model_name}_latency",
            values=latencies,
            metadata={'input_shape': input_shape, **self.system_info}
        )

    return results

@dataclass
class BenchmarkResult:
    metric_name: str
    values: List[float]
    metadata: Dict[str, Any] = field(default_factory=dict)

    def __post_init__(self):
        """Compute statistics after initialization."""
        if not self.values:
            raise ValueError("BenchmarkResult requires at least one measurement.")

        self.mean = statistics.mean(self.values)
        self.std = statistics.stdev(self.values) if len(self.values) > 1 else 0.0
        self.median = statistics.median(self.values)
        self.min_val = min(self.values)
        self.max_val = max(self.values)
        self.count = len(self.values)

        # 95% confidence interval for the mean
        if len(self.values) > 1:
            t_score = 1.96  # Approximate for large samples
            margin_error = t_score * (self.std / np.sqrt(self.count))
            self.ci_lower = self.mean - margin_error
            self.ci_upper = self.mean + margin_error
        else:
            self.ci_lower = self.ci_upper = self.mean

def __init__(self, models: List[Any], datasets: List[Any] = None,
             warmup_runs: int = DEFAULT_WARMUP_RUNS,
             measurement_runs: int = DEFAULT_MEASUREMENT_RUNS):
    self.models = models
    self.datasets = datasets or []
    self.warmup_runs = warmup_runs
    self.measurement_runs = measurement_runs

    # Capture system information for reproducibility
    self.system_info = {
        'platform': platform.platform(),
        'python_version': platform.python_version(),
        'cpu_count': os.cpu_count() or 1,
    }

# BAD: Only 3 measurements
benchmark = Benchmark(models, measurement_runs=3)  # Unreliable!

# GOOD: 10+ measurements
benchmark = Benchmark(models, measurement_runs=10)  # Statistical confidence

# Already handled in Benchmark class
for _ in range(self.warmup_runs):
    _ = model(input_data)  # Warmup (discarded)

for _ in range(self.measurement_runs):
    # Now measure steady-state performance

# Ensure all models use same input shape
results = benchmark.run_latency_benchmark(input_shape=(1, 28, 28))

# BenchmarkResult automatically computes confidence intervals
result = BenchmarkResult("latency", measurements)
print(f"{result.mean:.3f}ms ± {result.std:.3f}ms, 95% CI: [{result.ci_lower:.3f}, {result.ci_upper:.3f}]")

````{tab-item} Your TinyTorch
```python
from tinytorch.perf.benchmarking import Benchmark

# Setup models and benchmark
benchmark = Benchmark(
    models=[baseline_model, optimized_model],
    warmup_runs=5,
    measurement_runs=10
)

# Run latency benchmark
results = benchmark.run_latency_benchmark(input_shape=(1, 28, 28))

# Analyze results
for model_name, result in results.items():
    print(f"{model_name}: {result.mean*1000:.2f}ms ± {result.std*1000:.2f}ms")
```
````

````{tab-item} MLPerf (Industry Standard)
```python
import mlperf_loadgen as lg

# Configure benchmark scenario
settings = lg.TestSettings()
settings.scenario = lg.TestScenario.SingleStream
settings.mode = lg.TestMode.PerformanceOnly

# Run standardized benchmark
sut = SystemUnderTest(model)
lg.StartTest(sut, qsl, settings)

# Results include latency percentiles, throughput, accuracy
```
````

The statistical methodology, warmup protocols, and reproducibility requirements are identical. Production frameworks add automation, standardization across organizations, and hardware-specific optimizations. Understanding TinyTorch benchmarking gives you the foundation to work with any industry benchmarking framework.

:tags: [remove-input, remove-output]
from myst_nb import glue

# Production cost savings from latency reduction
scale_annual_cost = 50_000_000
scale_latency_reduction = 0.10
scale_savings = scale_annual_cost * scale_latency_reduction
glue("scale_savings", f"${scale_savings / 1_000_000:.0f} million")

:tags: [remove-input, remove-output]
import math

# Q1: Statistical Significance
# Baseline: mean=12.5ms, std=1.2ms, n=10
# Optimized: mean=11.8ms, std=1.5ms, n=10
q1_n = 10
q1_bl_mean = 12.5
q1_bl_std = 1.2
q1_bl_margin = 1.96 * (q1_bl_std / math.sqrt(q1_n))
q1_bl_ci_lo = q1_bl_mean - q1_bl_margin
q1_bl_ci_hi = q1_bl_mean + q1_bl_margin

q1_opt_mean = 11.8
q1_opt_std = 1.5
q1_opt_margin = 1.96 * (q1_opt_std / math.sqrt(q1_n))
q1_opt_ci_lo = q1_opt_mean - q1_opt_margin
q1_opt_ci_hi = q1_opt_mean + q1_opt_margin

q1_mean_diff = q1_bl_mean - q1_opt_mean

glue("q1_bl_margin", f"{q1_bl_margin:.2f}")
glue("q1_bl_ci_lo", f"{q1_bl_ci_lo:.2f}")
glue("q1_bl_ci_hi", f"{q1_bl_ci_hi:.2f}")
glue("q1_opt_margin", f"{q1_opt_margin:.2f}")
glue("q1_opt_ci_lo", f"{q1_opt_ci_lo:.2f}")
glue("q1_opt_ci_hi", f"{q1_opt_ci_hi:.2f}")
glue("q1_mean_diff", f"{q1_mean_diff:.1f}")

# Q2: Sample Size Calculation
# std=2.0ms, target margin=±0.5ms
q2_std = 2.0
q2_target = 0.5
q2_sqrt_n = 1.96 * q2_std / q2_target
q2_n = q2_sqrt_n ** 2
q2_n_ceil = math.ceil(q2_n)

glue("q2_sqrt_n", f"{q2_sqrt_n:.2f}")
glue("q2_n_raw", f"{q2_n:.1f}")
glue("q2_n_ceil", f"{q2_n_ceil}")

# Q3: Warmup Impact
# Without warmup measurements
q3_no_warmup = [15.2, 12.1, 10.8, 10.5, 10.6, 10.4]
q3_warmup = [10.5, 10.6, 10.4, 10.7, 10.5, 10.6]
q3_nw_mean = sum(q3_no_warmup) / len(q3_no_warmup)
q3_w_mean = sum(q3_warmup) / len(q3_warmup)
q3_latency_diff = q3_nw_mean - q3_w_mean
q3_latency_pct = q3_latency_diff / q3_nw_mean * 100

# Std values are given as approximate in the text (1.8 and 0.1)
q3_nw_std = 1.8
q3_w_std = 0.1
q3_var_reduction = (q3_nw_std - q3_w_std) / q3_nw_std * 100

glue("q3_nw_mean", f"{q3_nw_mean:.1f}ms")
glue("q3_w_mean", f"{q3_w_mean:.2f}ms")
glue("q3_latency_diff", f"{q3_latency_diff:.2f}ms")
glue("q3_latency_pct", f"{q3_latency_pct:.0f}%")
glue("q3_var_reduction", f"{q3_var_reduction:.0f}%")

# Q5: Measurement Overhead
# 1μs timer overhead, 50μs operation, 1000 measurements
q5_op_us = 50
q5_overhead_us = 1
q5_n_measurements = 1000
q5_total_op_us = q5_op_us * q5_n_measurements
q5_total_overhead_us = q5_overhead_us * q5_n_measurements
q5_total_op_ms = q5_total_op_us / 1000
q5_total_overhead_ms = q5_total_overhead_us / 1000
q5_total_measured_ms = q5_total_op_ms + q5_total_overhead_ms
q5_overhead_pct = (q5_total_overhead_ms / q5_total_measured_ms) * 100

glue("q5_total_op_us", f"{q5_total_op_us:,}")
glue("q5_total_op_ms", f"{q5_total_op_ms:.0f}ms")
glue("q5_total_overhead_us", f"{q5_total_overhead_us:,}")
glue("q5_total_overhead_ms", f"{q5_total_overhead_ms:.0f}ms")
glue("q5_total_measured_ms", f"{q5_total_measured_ms:.0f}ms")
glue("q5_overhead_pct", f"{q5_overhead_pct:.2f}%")

:class: dropdown

**Calculate 95% confidence intervals:**

Baseline: CI = mean ± 1.96 * (std / sqrt(n)) = 12.5 ± 1.96 * (1.2 / sqrt(10)) = 12.5 ± {glue:text}`q1_bl_margin` = [{glue:text}`q1_bl_ci_lo`, {glue:text}`q1_bl_ci_hi`]

Optimized: CI = 11.8 ± 1.96 * (1.5 / sqrt(10)) = 11.8 ± {glue:text}`q1_opt_margin` = [{glue:text}`q1_opt_ci_lo`, {glue:text}`q1_opt_ci_hi`]

**Result**: The confidence intervals OVERLAP (baseline goes as low as {glue:text}`q1_bl_ci_lo`, optimized goes as high as {glue:text}`q1_opt_ci_hi`). This means the difference is **NOT statistically significant** at the 95% confidence level. You cannot confidently claim the optimized model is faster.

**Lesson**: Always compute confidence intervals. A {glue:text}`q1_mean_diff`ms difference in means might seem meaningful, but with these variances and sample sizes, it could be random noise.

:class: dropdown

**Confidence interval formula**: margin = 1.96 * (std / sqrt(n))

**Solve for n**: 0.5 = 1.96 * (2.0 / sqrt(n))

sqrt(n) = 1.96 * 2.0 / 0.5 = {glue:text}`q2_sqrt_n`

n = {glue:text}`q2_sqrt_n`² = **{glue:text}`q2_n_raw` ≈ {glue:text}`q2_n_ceil` measurements**

**Lesson**: Achieving tight confidence intervals requires many measurements. Quadrupling precision (from ±1.0ms to ±0.5ms) requires 4x more samples (15 to 60). This is why professional benchmarks run hundreds of iterations.

:class: dropdown

**Without warmup:**
- Mean = (15.2 + 12.1 + 10.8 + 10.5 + 10.6 + 10.4) / 6 = **{glue:text}`q3_nw_mean`**
- Std = 1.8ms (high variance due to warmup effects)

**With warmup:**
- Mean = (10.5 + 10.6 + 10.4 + 10.7 + 10.5 + 10.6) / 6 = **{glue:text}`q3_w_mean`**
- Std = 0.1ms (low variance, stable measurements)

**Impact:**
- Latency reduced: {glue:text}`q3_nw_mean` - {glue:text}`q3_w_mean` = **{glue:text}`q3_latency_diff` ({glue:text}`q3_latency_pct` reduction)**
- Variance reduced: 1.8 → 0.1ms = **{glue:text}`q3_var_reduction` reduction in noise**

**Lesson**: Warmup eliminates cold-start effects and dramatically reduces measurement variance. Without warmup, you are measuring system startup behavior, not steady-state performance.

:class: dropdown

**Pareto-optimal definition**: A model is Pareto-optimal if no other model is better on ALL metrics simultaneously.

**Analysis:**
- Model A vs B: A is faster (5ms < 8ms) but less accurate (88% < 92%). Neither dominates. Both Pareto-optimal.
- Model A vs C: A is faster (5ms < 6ms) but less accurate (88% < 89%). Neither dominates. Both Pareto-optimal.
- Model B vs C: B is slower (8ms > 6ms) but more accurate (92% > 89%). Neither dominates. Both Pareto-optimal.

**Result**: **All three models are Pareto-optimal**. None is strictly dominated by another.

**Lesson**: The Pareto frontier shows trade-off options. If you need minimum latency, choose A. If you need maximum accuracy, choose B. If you want balanced performance, choose C.

:class: dropdown

**Total true operation time**: 50μs × 1000 = {glue:text}`q5_total_op_us`μs = {glue:text}`q5_total_op_ms`

**Total timer overhead**: 1μs × 1000 = {glue:text}`q5_total_overhead_us`μs = {glue:text}`q5_total_overhead_ms`

**Total measured time**: {glue:text}`q5_total_op_ms` + {glue:text}`q5_total_overhead_ms` = {glue:text}`q5_total_measured_ms`

**Overhead percentage**: ({glue:text}`q5_total_overhead_ms` / {glue:text}`q5_total_measured_ms`) × 100% = **{glue:text}`q5_overhead_pct`**

**Lesson**: Timer overhead is negligible for operations longer than ~50μs, but becomes significant for microsecond-scale operations. This is why we use `time.perf_counter()` with nanosecond resolution and minimal overhead. For operations under 10μs, consider measuring batches and averaging.

Apply everything you have learned in Modules 01-19 to compete in the TorchPerf Olympics! You will strategically combine optimization techniques, benchmark your results, and compete for the fastest, smallest, or most accurate model on the leaderboard.

- **[Launch Binder](https://mybinder.org/v2/gh/harvard-edge/cs249r_book/main?urlpath=lab/tree/tinytorch/modules/19_benchmarking/benchmarking.ipynb)** - Run interactively in browser, no setup required
- **[View Source](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/19_benchmarking/19_benchmarking.py)** - Browse the implementation code

Binder sessions are temporary. Download your completed notebook when done, or clone the repository for persistent local work.