The memory wall is the growing gap between how fast processors can compute and how fast memory can feed them data. For decades compute throughput (FLOP/s) grew far faster than memory bandwidth, so an increasing share of real workloads finish their math and then sit idle waiting for data. The wall is not a single component failing — it is a structural imbalance that makes bandwidth, not arithmetic, the binding constraint for more and more programs.\n\nCompute outran memory, and the gap compounds. Peak FLOP/s rose roughly an order of magnitude faster than DRAM bandwidth over successive generations. Because the two grow at different exponential rates, the ratio between them widens every year. The consequence is that a chip can have enormous nominal compute yet spend most cycles stalled, because the operands cannot arrive quickly enough — the useful throughput is set by the slower of the two, which is increasingly memory.\n\nThe roofline makes the wall precise. A kernel's arithmetic intensity — FLOPs performed per byte moved from memory — decides which limit binds. Left of the roofline's ridge point, performance is capped by bandwidth (memory-bound); right of it, by compute. As compute ceilings rise, the ridge point moves right, so more kernels fall into the bandwidth-bound region. Low-intensity operations that dominate modern AI inference — attention decode, matrix-vector products, elementwise ops — sit squarely in that region, limited by how fast weights and the KV cache stream from HBM.\n\n| Regime | Bound by | Example workloads | Fix targets |\n|---|---|---|---|\n| Left of ridge | memory bandwidth | LLM decode, GEMV, elementwise | move fewer bytes |\n| Right of ridge | compute (FLOPs) | large GEMM, training | more/faster FLOPs |\n| The trend | ridge moves right | more kernels go memory-bound | attack the byte side |\n\n``svg\n\n``\n\nEvery mitigation attacks the byte side, not the FLOP side. Since the shortage is bytes-per-second, the toolbox works to move fewer bytes or move them faster: larger on-chip caches and SRAM, stacked high-bandwidth memory (HBM), quantization to shrink each value, operator fusion to avoid round-trips to memory, and algorithms that keep data resident on-chip (FlashAttention's tiling, KV-cache compression, wafer-scale on-die memory). None of these add FLOPs; they raise effective arithmetic intensity so the same compute is less starved.\n\nRead the memory wall through a quant lens rather than a 'slow RAM' lens: it is the divergence between two exponentials — FLOP/s and bytes/s — and per the roofline that ratio, compared against a kernel's arithmetic intensity, tells you whether adding compute helps at all. For most AI inference the answer is no: the kernels are left of the ridge, so the design question is bytes-moved-per-useful-FLOP and how to lower it. Optimizing the memory wall means budgeting data movement, not chasing peak FLOPs that will sit idle.