````{grid} 1 1 3 3
:gutter: 3

```{grid-item-card} 🎧 Audio Overview

Listen to an AI-generated overview.

<audio controls style="width: 100%; height: 54px; margin-top: auto;">
  <source src="https://github.com/harvard-edge/cs249r_book/releases/download/tinytorch-audio-v0.1.1/15_quantization.mp3" type="audio/mpeg">
</audio>
```

```{grid-item-card} 🚀 Launch Binder

Run interactively in your browser.

<a href="https://mybinder.org/v2/gh/harvard-edge/cs249r_book/main?labpath=tinytorch%2Fmodules%2F15_quantization%2Fquantization.ipynb" target="_blank" style="display: flex; align-items: center; justify-content: center; width: 100%; height: 54px; margin-top: auto; background: #f97316; color: white; text-align: center; text-decoration: none; border-radius: 27px; font-size: 14px; box-sizing: border-box;">Open in Binder →</a>
```

```{grid-item-card} 📄 View Source

Browse the source code on GitHub.

<a href="https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/15_quantization/15_quantization.py" target="_blank" style="display: flex; align-items: center; justify-content: center; width: 100%; height: 54px; margin-top: auto; background: #14b8a6; color: white; text-align: center; text-decoration: none; border-radius: 27px; font-size: 14px; box-sizing: border-box;">View on GitHub →</a>
```

````

```{raw} html
<style>
.slide-viewer-container {
  margin: 0.5rem 0 1.5rem 0;
  background: #0f172a;
  border-radius: 1rem;
  overflow: hidden;
  box-shadow: 0 4px 20px rgba(0,0,0,0.15);
}
.slide-header {
  display: flex;
  align-items: center;
  justify-content: space-between;
  padding: 0.6rem 1rem;
  background: rgba(255,255,255,0.03);
}
.slide-title {
  display: flex;
  align-items: center;
  gap: 0.5rem;
  color: #94a3b8;
  font-weight: 500;
  font-size: 0.85rem;
}
.slide-subtitle {
  color: #64748b;
  font-weight: 400;
  font-size: 0.75rem;
}
.slide-toolbar {
  display: flex;
  align-items: center;
  gap: 0.375rem;
}
.slide-toolbar button {
  background: transparent;
  border: none;
  color: #64748b;
  width: 32px;
  height: 32px;
  border-radius: 0.375rem;
  cursor: pointer;
  font-size: 1.1rem;
  transition: all 0.15s;
  display: flex;
  align-items: center;
  justify-content: center;
}
.slide-toolbar button:hover {
  background: rgba(249, 115, 22, 0.15);
  color: #f97316;
}
.slide-nav-group {
  display: flex;
  align-items: center;
}
.slide-page-info {
  color: #64748b;
  font-size: 0.75rem;
  padding: 0 0.5rem;
  font-weight: 500;
}
.slide-zoom-group {
  display: flex;
  align-items: center;
  margin-left: 0.25rem;
  padding-left: 0.5rem;
  border-left: 1px solid rgba(255,255,255,0.1);
}
.slide-canvas-wrapper {
  display: flex;
  justify-content: center;
  align-items: center;
  padding: 0.5rem 1rem 1rem 1rem;
  min-height: 380px;
  background: #0f172a;
}
.slide-canvas {
  max-width: 100%;
  max-height: 350px;
  height: auto;
  border-radius: 0.5rem;
  box-shadow: 0 4px 24px rgba(0,0,0,0.4);
}
.slide-progress-wrapper {
  padding: 0 1rem 0.5rem 1rem;
}
.slide-progress-bar {
  height: 3px;
  background: rgba(255,255,255,0.08);
  border-radius: 1.5px;
  overflow: hidden;
  cursor: pointer;
}
.slide-progress-fill {
  height: 100%;
  background: #f97316;
  border-radius: 1.5px;
  transition: width 0.2s ease;
}
.slide-loading {
  color: #f97316;
  font-size: 0.9rem;
  display: flex;
  align-items: center;
  gap: 0.5rem;
}
.slide-loading::before {
  content: '';
  width: 18px;
  height: 18px;
  border: 2px solid rgba(249, 115, 22, 0.2);
  border-top-color: #f97316;
  border-radius: 50%;
  animation: slide-spin 0.8s linear infinite;
}
@keyframes slide-spin {
  to { transform: rotate(360deg); }
}
.slide-footer {
  display: flex;
  justify-content: center;
  gap: 0.5rem;
  padding: 0.6rem 1rem;
  background: rgba(255,255,255,0.02);
  border-top: 1px solid rgba(255,255,255,0.05);
}
.slide-footer a {
  display: inline-flex;
  align-items: center;
  gap: 0.375rem;
  background: #f97316;
  color: white;
  padding: 0.4rem 0.9rem;
  border-radius: 2rem;
  text-decoration: none;
  font-weight: 500;
  font-size: 0.75rem;
  transition: all 0.15s;
}
.slide-footer a:hover {
  background: #ea580c;
  color: white;
}
.slide-footer a.secondary {
  background: transparent;
  color: #94a3b8;
  border: 1px solid rgba(255,255,255,0.15);
}
.slide-footer a.secondary:hover {
  background: rgba(255,255,255,0.05);
  color: #f8fafc;
}
@media (max-width: 600px) {
  .slide-header { flex-direction: column; gap: 0.5rem; padding: 0.5rem 0.75rem; }
  .slide-toolbar button { width: 28px; height: 28px; }
  .slide-canvas-wrapper { min-height: 260px; padding: 0.5rem; }
  .slide-canvas { max-height: 220px; }
}
</style>

<div class="slide-viewer-container" id="slide-viewer-15_quantization">
  <div class="slide-header">
    <div class="slide-title">
      <span>🔥</span>
      <span>Slide Deck</span>
      <span class="slide-subtitle">· AI-generated</span>
    </div>
    <div class="slide-toolbar">
      <div class="slide-nav-group">
        <button onclick="slideNav('15_quantization', -1)" title="Previous">‹</button>
        <span class="slide-page-info"><span id="slide-num-15_quantization">1</span> / <span id="slide-count-15_quantization">-</span></span>
        <button onclick="slideNav('15_quantization', 1)" title="Next">›</button>
      </div>
      <div class="slide-zoom-group">
        <button onclick="slideZoom('15_quantization', -0.25)" title="Zoom out">−</button>
        <button onclick="slideZoom('15_quantization', 0.25)" title="Zoom in">+</button>
      </div>
    </div>
  </div>
  <div class="slide-canvas-wrapper">
    <div id="slide-loading-15_quantization" class="slide-loading">Loading slides...</div>
    <canvas id="slide-canvas-15_quantization" class="slide-canvas" style="display:none;"></canvas>
  </div>
  <div class="slide-progress-wrapper">
    <div class="slide-progress-bar" onclick="slideProgress('15_quantization', event)">
      <div class="slide-progress-fill" id="slide-progress-15_quantization" style="width: 0%;"></div>
    </div>
  </div>
  <div class="slide-footer">
    <a href="../_static/slides/15_quantization.pdf" download>⬇ Download</a>
    <a href="#" onclick="slideFullscreen('15_quantization'); return false;" class="secondary">⛶ Fullscreen</a>
  </div>
</div>

<script src="https://cdnjs.cloudflare.com/ajax/libs/pdf.js/3.11.174/pdf.min.js"></script>
<script>
(function() {
  if (window.slideViewersInitialized) return;
  window.slideViewersInitialized = true;

  pdfjsLib.GlobalWorkerOptions.workerSrc = 'https://cdnjs.cloudflare.com/ajax/libs/pdf.js/3.11.174/pdf.worker.min.js';

  window.slideViewers = {};

  window.initSlideViewer = function(id, pdfUrl) {
    const viewer = { pdf: null, page: 1, scale: 1.3, rendering: false, pending: null };
    window.slideViewers[id] = viewer;

    const canvas = document.getElementById('slide-canvas-' + id);
    const ctx = canvas.getContext('2d');

    function render(num) {
      viewer.rendering = true;
      viewer.pdf.getPage(num).then(function(page) {
        const viewport = page.getViewport({scale: viewer.scale});
        canvas.height = viewport.height;
        canvas.width = viewport.width;
        page.render({canvasContext: ctx, viewport: viewport}).promise.then(function() {
          viewer.rendering = false;
          if (viewer.pending !== null) { render(viewer.pending); viewer.pending = null; }
        });
      });
      document.getElementById('slide-num-' + id).textContent = num;
      document.getElementById('slide-progress-' + id).style.width = (num / viewer.pdf.numPages * 100) + '%';
    }

    function queue(num) { if (viewer.rendering) viewer.pending = num; else render(num); }

    pdfjsLib.getDocument(pdfUrl).promise.then(function(pdf) {
      viewer.pdf = pdf;
      document.getElementById('slide-count-' + id).textContent = pdf.numPages;
      document.getElementById('slide-loading-' + id).style.display = 'none';
      canvas.style.display = 'block';
      render(1);
    }).catch(function() {
      document.getElementById('slide-loading-' + id).innerHTML = 'Unable to load. <a href="' + pdfUrl + '" style="color:#f97316;">Download PDF</a>';
    });

    viewer.queue = queue;
  };

  window.slideNav = function(id, dir) {
    const v = window.slideViewers[id];
    if (!v || !v.pdf) return;
    const newPage = v.page + dir;
    if (newPage >= 1 && newPage <= v.pdf.numPages) { v.page = newPage; v.queue(newPage); }
  };

  window.slideZoom = function(id, delta) {
    const v = window.slideViewers[id];
    if (!v) return;
    v.scale = Math.max(0.5, Math.min(3, v.scale + delta));
    v.queue(v.page);
  };

  window.slideProgress = function(id, event) {
    const v = window.slideViewers[id];
    if (!v || !v.pdf) return;
    const bar = event.currentTarget;
    const pct = (event.clientX - bar.getBoundingClientRect().left) / bar.offsetWidth;
    const newPage = Math.max(1, Math.min(v.pdf.numPages, Math.ceil(pct * v.pdf.numPages)));
    if (newPage !== v.page) { v.page = newPage; v.queue(newPage); }
  };

  window.slideFullscreen = function(id) {
    const el = document.getElementById('slide-viewer-' + id);
    if (el.requestFullscreen) el.requestFullscreen();
    else if (el.webkitRequestFullscreen) el.webkitRequestFullscreen();
  };
})();

initSlideViewer('15_quantization', '../_static/slides/15_quantization.pdf');
</script>
```

:tags: [remove-input, remove-output]
from myst_nb import glue

# Model sizes in FP32 (4 bytes per parameter)
bert_params = 110_000_000
gpt2_params = 1_500_000_000
gpt3_params = 175_000_000_000
bert_mb = bert_params * 4 / 1024**2
gpt2_gb = gpt2_params * 4 / 1024**3
gpt3_gb = gpt3_params * 4 / 1024**3
glue("bert_mb", f"{bert_mb:.0f} MB")
glue("gpt2_gb", f"{gpt2_gb:.1f} GB")
glue("gpt3_gb", f"{gpt3_gb:.0f} GB")

# Quantized BERT (INT8 = 1 byte per param)
bert_int8_mb = bert_params * 1 / 1024**2
glue("bert_int8_mb", f"{bert_int8_mb:.0f} MB")

- **Implement** INT8 quantization with symmetric scaling and zero-point calculation for 4× memory reduction
- **Master** calibration techniques that optimize quantization parameters using sample data distributions
- **Understand** quantization error propagation and accuracy preservation strategies in compressed models
- **Connect** your implementation to production frameworks like TensorFlow Lite and PyTorch quantization APIs
- **Analyze** memory-accuracy trade-offs across different quantization strategies and model architectures

:align: center
:caption: Quantization System
flowchart TB
    subgraph "Quantization System"
        A["quantize_int8()
FP32 → INT8 conversion"]
        B["dequantize_int8()
INT8 → FP32 restoration"]
        C["QuantizedLinear
Quantized layer class"]
        D["quantize_model()
Full network quantization"]
        E["Calibration
Parameter optimization"]
    end

    A --> C
    B --> C
    C --> D
    E --> D

    style A fill:#e1f5ff
    style B fill:#fff3cd
    style C fill:#f8d7da
    style D fill:#d4edda
    style E fill:#e2d5f1

# Compress a 400MB model to 100MB
quantize_model(model, calibration_data=sample_inputs)
# Now model uses 4× less memory with <1% accuracy loss

quantize_int8(tensor: Tensor) -> Tuple[Tensor, float, int]

dequantize_int8(q_tensor: Tensor, scale: float, zero_point: int) -> Tensor

Quantizer()

def quantize_int8(tensor: Tensor) -> Tuple[Tensor, float, int]:
    """Quantize FP32 tensor to INT8 using asymmetric quantization."""
    data = tensor.data

    # Step 1: Find dynamic range
    min_val = float(np.min(data))
    max_val = float(np.max(data))

    # Step 2: Handle edge case (constant tensor)
    if abs(max_val - min_val) < EPSILON:
        scale = 1.0
        zero_point = 0
        quantized_data = np.zeros_like(data, dtype=np.int8)
        return Tensor(quantized_data), scale, zero_point

    # Step 3: Calculate scale and zero_point
    scale = (max_val - min_val) / (INT8_RANGE - 1)
    zero_point = int(np.round(INT8_MIN_VALUE - min_val / scale))
    zero_point = int(np.clip(zero_point, INT8_MIN_VALUE, INT8_MAX_VALUE))

    # Step 4: Apply quantization formula
    quantized_data = np.round(data / scale + zero_point)
    quantized_data = np.clip(quantized_data, INT8_MIN_VALUE, INT8_MAX_VALUE).astype(np.int8)

    return Tensor(quantized_data), scale, zero_point

def dequantize_int8(q_tensor: Tensor, scale: float, zero_point: int) -> Tensor:
    """Dequantize INT8 tensor back to FP32."""
    dequantized_data = (q_tensor.data.astype(np.float32) - zero_point) * scale
    return Tensor(dequantized_data)

class QuantizedLinear:
    """Quantized version of Linear layer using INT8 arithmetic."""

    def __init__(self, linear_layer: Linear):
        """Create quantized version of existing linear layer."""
        self.original_layer = linear_layer

        # Quantize weights
        self.q_weight, self.weight_scale, self.weight_zero_point = quantize_int8(linear_layer.weight)

        # Quantize bias if it exists
        if linear_layer.bias is not None:
            self.q_bias, self.bias_scale, self.bias_zero_point = quantize_int8(linear_layer.bias)
        else:
            self.q_bias = None
            self.bias_scale = None
            self.bias_zero_point = None

        # Store input quantization parameters (set during calibration)
        self.input_scale = None
        self.input_zero_point = None

def forward(self, x: Tensor) -> Tensor:
    """Forward pass with quantized computation."""
    # Dequantize weights
    weight_fp32 = dequantize_int8(self.q_weight, self.weight_scale, self.weight_zero_point)

    # Perform computation (same as original layer)
    result = x.matmul(weight_fp32)

    # Add bias if it exists
    if self.q_bias is not None:
        bias_fp32 = dequantize_int8(self.q_bias, self.bias_scale, self.bias_zero_point)
        result = Tensor(result.data + bias_fp32.data)

    return result

def calibrate(self, sample_inputs: List[Tensor]):
    """Calibrate input quantization parameters using sample data."""
    # Collect all input values
    all_values = []
    for inp in sample_inputs:
        all_values.extend(inp.data.flatten())

    all_values = np.array(all_values)

    # Calculate input quantization parameters
    min_val = float(np.min(all_values))
    max_val = float(np.max(all_values))

    if abs(max_val - min_val) < EPSILON:
        self.input_scale = 1.0
        self.input_zero_point = 0
    else:
        self.input_scale = (max_val - min_val) / (INT8_RANGE - 1)
        self.input_zero_point = int(np.round(INT8_MIN_VALUE - min_val / self.input_scale))
        self.input_zero_point = np.clip(self.input_zero_point, INT8_MIN_VALUE, INT8_MAX_VALUE)

````{tab-item} Your TinyTorch
```python
from tinytorch.perf.quantization import quantize_model, QuantizedLinear
from tinytorch.core.layers import Linear, Sequential

# Create model
model = Sequential(
    Linear(784, 128),
    Linear(128, 10)
)

# Quantize to INT8
calibration_data = [sample_batch1, sample_batch2, ...]
quantize_model(model, calibration_data)

# Use quantized model
output = model.forward(x)  # 4× less memory!
```
````

````{tab-item} ⚡ PyTorch
```python
import torch
import torch.quantization as quantization

# Create model
model = torch.nn.Sequential(
    torch.nn.Linear(784, 128),
    torch.nn.Linear(128, 10)
)

# Quantize to INT8
model.qconfig = quantization.get_default_qconfig('fbgemm')
model_prepared = quantization.prepare(model)
# Run calibration
for batch in calibration_data:
    model_prepared(batch)
model_quantized = quantization.convert(model_prepared)

# Use quantized model
output = model_quantized(x)  # 4× less memory!
```
````

The core quantization mathematics: scale calculation, zero-point mapping, INT8 range clipping. When you debug PyTorch quantization errors, you'll understand exactly what's happening because you implemented the same algorithms.

:tags: [remove-input, remove-output]
import math

# Q1: 3-layer network parameter counting
q1_l1 = 784 * 256 + 256
q1_l2 = 256 * 128 + 128
q1_l3 = 128 * 10 + 10
q1_total = q1_l1 + q1_l2 + q1_l3
q1_fp32_bytes = q1_total * 4
q1_int8_bytes = q1_total * 1
q1_savings = q1_fp32_bytes - q1_int8_bytes
glue("q1_l1", f"{q1_l1:,}")
glue("q1_l2", f"{q1_l2:,}")
glue("q1_l3", f"{q1_l3:,}")
glue("q1_total", f"{q1_total:,}")
glue("q1_fp32_bytes", f"{q1_fp32_bytes:,}")
glue("q1_fp32_mb", f"{q1_fp32_bytes / 1024**2:.2f} MB")
glue("q1_int8_bytes", f"{q1_int8_bytes:,}")
glue("q1_int8_mb", f"{q1_int8_bytes / 1024**2:.2f} MB")
glue("q1_savings_mb", f"{q1_savings / 1024**2:.2f} MB")

# Q2: Quantization error and SNR
q2_range = 1.0
q2_levels = 255
q2_scale = q2_range / q2_levels
q2_max_error = q2_scale / 2
q2_snr = 20 * math.log10(q2_range / q2_scale)
glue("q2_scale", f"{q2_scale:.6f}")
glue("q2_max_error", f"±{q2_max_error:.6f}")
glue("q2_snr", f"{q2_snr:.0f} dB")

# Q4: Loading time
q4_fp32_mb = 100_000_000 * 4 / 1024**2
q4_int8_mb = 100_000_000 * 1 / 1024**2
q4_bandwidth = 500  # MB/s
q4_fp32_time = q4_fp32_mb / q4_bandwidth
q4_int8_time = q4_int8_mb / q4_bandwidth
glue("q4_fp32_mb", f"{q4_fp32_mb:.0f} MB")
glue("q4_int8_mb", f"{q4_int8_mb:.0f} MB")
glue("q4_fp32_time", f"{q4_fp32_time:.1f} seconds")
glue("q4_int8_time", f"{q4_int8_time:.2f} seconds")
glue("q4_time_saved", f"{q4_fp32_time - q4_int8_time:.1f}s")

# Q5: SIMD register capacity
simd_bits = 512
fp32_per_reg = simd_bits // 32
int8_per_reg = simd_bits // 8
glue("q5_fp32", f"{fp32_per_reg}")
glue("q5_int8", f"{int8_per_reg}")
glue("q5_ratio", f"{int8_per_reg // fp32_per_reg}×")

:class: dropdown

**Parameter count:**
- Layer 1: (784 × 256) + 256 = {glue:text}`q1_l1`
- Layer 2: (256 × 128) + 128 = {glue:text}`q1_l2`
- Layer 3: (128 × 10) + 10 = {glue:text}`q1_l3`
- **Total: {glue:text}`q1_total` parameters**

**Memory usage:**
- FP32: {glue:text}`q1_total` × 4 bytes = **{glue:text}`q1_fp32_bytes` bytes ≈ {glue:text}`q1_fp32_mb`**
- INT8: {glue:text}`q1_total` × 1 byte = **{glue:text}`q1_int8_bytes` bytes ≈ {glue:text}`q1_int8_mb`**
- **Savings: {glue:text}`q1_savings_mb` (75% reduction, 4× compression)**

This shows why quantization matters: even small models benefit significantly.

:class: dropdown

**Quantization error:**
- Range: 0.5 - (-0.5) = 1.0
- Scale: 1.0 / 255 = **{glue:text}`q2_scale`**
- Max error: scale / 2 = **{glue:text}`q2_max_error`** (half step size)

**Signal-to-noise ratio:**
- SNR = 20 × log₁₀(signal_range / quantization_step)
- SNR = 20 × log₁₀(1.0 / {glue:text}`q2_scale`)
- SNR = 20 × log₁₀(255)
- SNR ≈ **{glue:text}`q2_snr`**

This is sufficient for neural networks (typical requirement: >40 dB). The 8-bit quantization provides approximately 6 dB per bit, matching the theoretical limit.

:class: dropdown

**Recommended: 100-1000 samples** (typically 500)

**Reasoning:**
- **Too few (<100)**: Risk missing outliers, suboptimal scale/zero-point
- **Too many (>1000)**: Diminishing returns, calibration time wasted
- **Sweet spot (100-1000)**: Captures distribution, fast calibration

**Trade-off analysis:**
- 10 samples: Fast (1 second), but might miss distribution tails → poor accuracy
- 100 samples: Medium (5 seconds), good representation → 98% accuracy
- 1000 samples: Slow (30 seconds), comprehensive → 98.5% accuracy
- 10000 samples: Very slow (5 minutes), overkill → 98.6% accuracy

**Conclusion**: Calibration accuracy plateaus around 100-1000 samples. Use more only if accuracy is critical (medical, autonomous vehicles).

:class: dropdown

**Loading time:**
- FP32 size: 100M × 4 bytes = {glue:text}`q4_fp32_mb`
- INT8 size: 100M × 1 byte = {glue:text}`q4_int8_mb`
- FP32 load time: {glue:text}`q4_fp32_mb` / 500 MB/s = **{glue:text}`q4_fp32_time`**
- INT8 load time: {glue:text}`q4_int8_mb` / 500 MB/s = **{glue:text}`q4_int8_time`**
- **Speedup: 4× faster loading**

**User experience impact:**
- Mobile app launch: {glue:text}`q4_fp32_time` → {glue:text}`q4_int8_time` (**{glue:text}`q4_time_saved` faster startup**)
- Cloud inference: {glue:text}`q4_fp32_time` latency → {glue:text}`q4_int8_time` latency (**4× better throughput**)
- Model updates: {glue:text}`q4_fp32_mb` download → {glue:text}`q4_int8_mb` download (**75% less data usage**)

**Key insight**: Quantization reduces not just RAM usage, but also disk I/O, network transfer, and cold-start latency. The 4× reduction applies to all memory movement operations.

:class: dropdown

**SIMD capacity:**
- 512-bit register with FP32: 512 / 32 = **{glue:text}`q5_fp32` values**
- 512-bit register with INT8: 512 / 8 = **{glue:text}`q5_int8` values**
- **Theoretical speedup: {glue:text}`q5_int8`/{glue:text}`q5_fp32` = {glue:text}`q5_ratio`**

**Why actual speedup is 2-3× (not 4×):**

1. **Dequantization overhead**: Converting INT8 → FP32 for activations takes time
2. **Memory bandwidth bottleneck**: INT8 ops are so fast, memory can't feed data fast enough
3. **Mixed precision**: Activations often stay FP32, only weights quantized
4. **Non-compute operations**: Batch norm, softmax, etc. remain FP32 (can't quantize easily)

**Real-world speedup breakdown:**
- Compute-bound workloads (large matmuls): **3-4× speedup**
- Memory-bound workloads (small layers): **1.5-2× speedup**
- Typical mixed models: **2-3× average speedup**

**Key insight**: INT8 quantization shines when matrix multiplications dominate (transformers, large MLPs). For convolutional layers with small kernels, memory bandwidth limits speedup.

Implement model pruning and weight compression techniques. You'll build structured pruning that removes entire neurons and channels, achieving 2-10× speedup by reducing computation, not just memory.

- **[Launch Binder](https://mybinder.org/v2/gh/harvard-edge/cs249r_book/main?urlpath=lab/tree/tinytorch/modules/15_quantization/quantization.ipynb)** - Run interactively in browser, no setup required
- **[View Source](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/15_quantization/15_quantization.py)** - Browse the implementation code

Binder sessions are temporary. Download your completed notebook when done, or clone the repository for persistent local work.