Memory hierarchy is the layered organization of storage in a computer system — from tiny, ultra-fast registers inside the processor core, through multiple levels of SRAM cache, to capacious but slower DRAM, and finally to non-volatile storage. The gap between processor compute throughput and memory bandwidth — the memory wall — is the dominant performance bottleneck in modern AI workloads.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
<rect width="760" height="470" fill="#0d1117"/>
<text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Memory Hierarchy — Speed vs Capacity Tradeoff</text>
<text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">Layered storage from registers to cloud storage, exploiting temporal and spatial locality</text>

<!-- Main container -->
<rect x="25" y="70" width="710" height="330" rx="12" fill="#080d14" stroke="#233043" stroke-width="1"/>

<!-- Pyramid structure (fast small on left, slow large on right) -->
<!-- Registers (top-left: fastest, smallest) -->
<rect x="45" y="90" width="65" height="50" rx="4" fill="#2a1a1a" stroke="#f87171" stroke-width="1.2"/>
<text x="77" y="107" fill="#fca5a5" font-size="10" font-weight="600" text-anchor="middle">Registers</text>
<text x="77" y="122" fill="#8b98a5" font-size="8" text-anchor="middle">0 cyc · 1 KB</text>
<text x="77" y="133" fill="#6b7684" font-size="7.5" text-anchor="middle">~∞ GB/s</text>

<!-- L1 Cache -->
<rect x="120" y="90" width="85" height="50" rx="4" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.2"/>
<text x="162" y="107" fill="#fbbf24" font-size="10" font-weight="600" text-anchor="middle">L1 Cache</text>
<text x="162" y="122" fill="#8b98a5" font-size="8" text-anchor="middle">4 cyc · 64 KB</text>
<text x="162" y="133" fill="#6b7684" font-size="7.5" text-anchor="middle">~1 TB/s</text>

<!-- L2 Cache -->
<rect x="215" y="75" width="95" height="65" rx="4" fill="#14261f" stroke="#34d399" stroke-width="1.2"/>
<text x="262" y="100" fill="#6ee7b7" font-size="10" font-weight="600" text-anchor="middle">L2 Cache</text>
<text x="262" y="115" fill="#8b98a5" font-size="8" text-anchor="middle">12 cyc · 2 MB</text>
<text x="262" y="126" fill="#6b7684" font-size="7.5" text-anchor="middle">~500 GB/s</text>

<!-- L3 Cache (shared) -->
<rect x="320" y="60" width="105" height="80" rx="4" fill="#1a1520" stroke="#a78bfa" stroke-width="1.3"/>
<text x="372" y="090" fill="#c4b5fd" font-size="10" font-weight="600" text-anchor="middle">L3 Cache</text>
<text x="372" y="105" fill="#8b98a5" font-size="8" text-anchor="middle">40 cyc · 32 MB</text>
<text x="372" y="116" fill="#6b7684" font-size="7.5" text-anchor="middle">~200 GB/s (shared)</text>

<!-- HBM (AI accelerators) -->
<rect x="435" y="65" width="120" height="75" rx="4" fill="#0f1a3a" stroke="#60a5fa" stroke-width="1.3"/>
<text x="495" y="092" fill="#93c5fd" font-size="10" font-weight="600" text-anchor="middle">HBM / HBM3E</text>
<text x="495" y="107" fill="#8b98a5" font-size="8" text-anchor="middle">100 cyc · 192 GB</text>
<text x="495" y="118" fill="#6b7684" font-size="7.5" text-anchor="middle">2–8 TB/s (GPU/TPU)</text>

<!-- DDR DRAM (bottom-right: slowest off-package, largest capacity) -->
<rect x="565" y="85" width="130" height="55" rx="4" fill="#0b1520" stroke="#3b82f6" stroke-width="1.2"/>
<text x="630" y="105" fill="#60a5fa" font-size="10" font-weight="600" text-anchor="middle">DDR5 DRAM</text>
<text x="630" y="120" fill="#8b98a5" font-size="8" text-anchor="middle">~50 ns · 512 GB</text>
<text x="630" y="131" fill="#6b7684" font-size="7.5" text-anchor="middle">50–100 GB/s</text>

<!-- Access time scale (vertical axis) -->
<text x="35" y="160" fill="#8b98a5" font-size="9" text-anchor="end" font-weight="600">Latency (cycles)</text>
<line x1="40" y1="170" x2="40" y2="330" stroke="#233043" stroke-width="1"/>
<text x="38" y="175" fill="#6b7684" font-size="8" text-anchor="end">0</text>
<text x="38" y="220" fill="#6b7684" font-size="8" text-anchor="end">50</text>
<text x="38" y="265" fill="#6b7684" font-size="8" text-anchor="end">500</text>
<text x="38" y="335" fill="#6b7684" font-size="8" text-anchor="end">10K+</text>

<!-- Capacity scale labels (horizontal axis) -->
<line x1="40" y1="350" x2="710" y2="350" stroke="#233043" stroke-width="1"/>
<text x="45" y="365" fill="#6b7684" font-size="8">KB</text>
<text x="250" y="365" fill="#6b7684" font-size="8">MB</text>
<text x="620" y="365" fill="#6b7684" font-size="8">GB</text>

<!-- Storage tiers (SSD, Object Store, etc.) -->
<rect x="45" y="160" width="650" height="28" rx="3" fill="#1a2a2a" stroke="#4a6a6a" stroke-width="1"/>
<text x="65" y="177" fill="#6a9a9a" font-size="9" font-weight="600">NVMe SSD (10 µs)</text>
<text x="250" y="177" fill="#6a8080" font-size="9" font-weight="600">Cloud Object Store (~1 ms)</text>

<!-- Hit rate annotation -->
<rect x="45" y="200" width="650" height="60" rx="4" fill="#1a1a2a" stroke="#3a3a6a" stroke-width="1"/>
<text x="380" y="218" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">THE LOCALITY ILLUSION</text>
<text x="60" y="238" fill="#8b98a5" font-size="8.5">T_avg = T_hit + (1−h) × T_miss — temporal locality (data reused) + spatial locality (sequential access)</text>
<text x="60" y="252" fill="#8b98a5" font-size="8.5">95%+ of CPU accesses hit L1/L2 → average latency ~1–3 ns, not DRAM's ~50 ns. Cache coherence and TLBs</text>
<text x="60" y="266" fill="#6b7684" font-size="8">manage sharing across cores. AI accelerators trade latency optimization for bandwidth: HBM + tiny SRAM tilings.</text>

<!-- Memory wall note -->
<rect x="45" y="270" width="650" height="60" rx="4" fill="#0f1a0f" stroke="#2f6d55" stroke-width="1"/>
<text x="380" y="288" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">THE MEMORY WALL</text>
<text x="60" y="308" fill="#8b98a5" font-size="8.5">Processor speed: 1000× since 1980s. DRAM latency: 2× since 1980s. The gap widens every generation.</text>
<text x="60" y="322" fill="#6b7684" font-size="8">Frontier AI models bottleneck on memory bandwidth, not compute. HBM3E, stacking, and systolic array tilings</text>
<text x="60" y="334" fill="#6b7684" font-size="8">attempt to close the gap—but silicon supply constrains HBM capacity growth.</text>

<!-- Footer -->
<text x="380" y="450" fill="#6b7684" font-size="11" text-anchor="middle">CFS Systolic/HBM simulators model tiling and bandwidth tradeoffs for your target accelerator architecture.</text>
</svg>

Locality and reuse — caches work because programs exhibit temporal locality (same data reused soon) and spatial locality (nearby data accessed together). A cache line is 64 bytes; a miss fetches the whole line from the next level up. Prefetching and out-of-order execution hide latency in software-transparent ways, but only when access patterns are predictable.

The 3C miss model organizes misses by root cause: compulsory misses hit cold data on first access; capacity misses occur when the working set exceeds the cache size; conflict misses arise from limited set-associativity. The 3C model guides micro-architects when balancing cache size vs die area — larger caches reduce capacity misses but increase latency and leakage.

The memory wall in AI — a matrix-multiply on an H100 peaks at ~989 TFLOP/s FP16, but HBM3 delivers only 3.35 TB/s. The arithmetic-intensity ridge point is ~295 FLOP/byte. LLM decode steps (attending over a KV cache) have arithmetic intensity of ~1–3 FLOP/byte, placing them deep in the memory-bound region. GEMM-heavy prefill operations are near or above the ridge and are compute-bound.

HBM vs DRAM vs SRAM — HBM stacks DRAM dies vertically on an interposer alongside the GPU die, delivering 10–20× DDR5 bandwidth at higher cost and lower capacity per dollar. On-chip SRAM (L1–L3 and GPU shared memory) is 50–100× faster still but costs ~100× more per bit of capacity. 3D-IC and chiplet designs are pulling larger SRAM pools physically closer to compute to close the wall.

Quantization and kernel tiling — reducing weight precision from FP32 to FP8 cuts memory traffic 4×, shifting the ridge point left and turning more operations compute-bound. Kernel tiling (FlashAttention, Triton, CUDA shared memory) explicitly manages SRAM as a software-controlled scratchpad, replacing round-trips through HBM with on-chip streaming. These techniques are now core competencies for AI chip and systems engineers.

Read the memory hierarchy through a locality-and-reuse lens rather than a speed-numbers lens: the hierarchy exists to exploit program structure, not merely to offer faster storage at each level — every cache miss is a signal that the access pattern is fighting the hardware's assumptions.