memory hierarchy
**Memory hierarchy** is the layered organization of storage in a computer system — from tiny, ultra-fast registers inside the processor core, through multiple levels of SRAM cache, to capacious but slower DRAM, and finally to non-volatile storage. The gap between processor compute throughput and memory bandwidth — the **memory wall** — is the dominant performance bottleneck in modern AI workloads.
```svg
```
| Level | Capacity | Latency | Bandwidth |
|---|---|---|---|
| Registers | < 1 KB | < 1 ns | > 10 TB/s |
| L1 cache | 32–64 KB | 1–4 ns | ~3 TB/s |
| L2 cache | 256 KB–4 MB | 5–15 ns | ~1 TB/s |
| L3 cache | 8–64 MB | 20–50 ns | ~500 GB/s |
| HBM3 (AI GPU) | 80–192 GB | ~80 ns | 3.35 TB/s |
| DDR5 DRAM | 16–512 GB | 60–100 ns | ~100 GB/s |
| NVMe SSD | 1–8 TB | 50–200 µs | ~12 GB/s |
**Locality and reuse** — caches work because programs exhibit temporal locality (same data reused soon) and spatial locality (nearby data accessed together). A cache line is 64 bytes; a miss fetches the whole line from the next level up. Prefetching and out-of-order execution hide latency in software-transparent ways, but only when access patterns are predictable.
**The 3C miss model** organizes misses by root cause: compulsory misses hit cold data on first access; capacity misses occur when the working set exceeds the cache size; conflict misses arise from limited set-associativity. The 3C model guides micro-architects when balancing cache size vs die area — larger caches reduce capacity misses but increase latency and leakage.
**The memory wall in AI** — a matrix-multiply on an H100 peaks at ~989 TFLOP/s FP16, but HBM3 delivers only 3.35 TB/s. The arithmetic-intensity ridge point is ~295 FLOP/byte. LLM decode steps (attending over a KV cache) have arithmetic intensity of ~1–3 FLOP/byte, placing them deep in the memory-bound region. GEMM-heavy prefill operations are near or above the ridge and are compute-bound.
**HBM vs DRAM vs SRAM** — HBM stacks DRAM dies vertically on an interposer alongside the GPU die, delivering 10–20× DDR5 bandwidth at higher cost and lower capacity per dollar. On-chip SRAM (L1–L3 and GPU shared memory) is 50–100× faster still but costs ~100× more per bit of capacity. 3D-IC and chiplet designs are pulling larger SRAM pools physically closer to compute to close the wall.
**Quantization and kernel tiling** — reducing weight precision from FP32 to FP8 cuts memory traffic 4×, shifting the ridge point left and turning more operations compute-bound. Kernel tiling (FlashAttention, Triton, CUDA shared memory) explicitly manages SRAM as a software-controlled scratchpad, replacing round-trips through HBM with on-chip streaming. These techniques are now core competencies for AI chip and systems engineers.
Read the memory hierarchy through a **locality-and-reuse lens rather than a speed-numbers lens**: the hierarchy exists to exploit program structure, not merely to offer faster storage at each level — every cache miss is a signal that the access pattern is fighting the hardware's assumptions.