hbm

High-bandwidth memory (HBM) is stacked DRAM built to feed GPUs and AI accelerators with far more bandwidth per package edge and per watt than any planar memory can deliver.\n\n**The stack is vertical, and that is the whole idea.** Four to sixteen DRAM dies are thinned and stacked on a base/logic die, wired together by thousands of through-silicon vias (TSVs). That creates a 1024-bit interface per stack (2048-bit in HBM4) at a modest per-pin clock. Several stacks sit beside the accelerator on a silicon interposer, so the memory bus never has to leave the package.\n\n**Bandwidth is why it exists.** Training streams large tensors continuously; autoregressive inference re-reads model weights and the KV cache on every token. Both are memory-bound, so aggregate bytes-per-second — not clock speed — sets throughput. HBM buys bandwidth the honest way: make the bus enormously wide and keep each wire slow, so energy per bit stays low even as total bandwidth climbs into the TB/s.\n\n| Generation | Per-pin rate | Bus width | Bandwidth / stack | Max stack | Capacity / stack | Era |\n|---|---|---|---|---|---|---|\n| HBM2 | ~2.4 Gb/s | 1024-bit | ~0.31 TB/s | 8-Hi | up to 8 GB | V100 |\n| HBM2E | ~3.6 Gb/s | 1024-bit | ~0.46 TB/s | 8-Hi | up to 16 GB | A100 |\n| HBM3 | ~6.4 Gb/s | 1024-bit | ~0.82 TB/s | 12-Hi | up to 24 GB | H100 |\n| HBM3E | ~9.6 Gb/s | 1024-bit | ~1.2 TB/s | 12-Hi | up to 36 GB | H200 / B200 |\n| HBM4 | ~8 Gb/s | 2048-bit | ~1.6-2.0 TB/s | 16-Hi | 48-64 GB | next-gen |\n\n```svg\n\n```\n\n**HBM is inseparable from packaging.** The stack, interposer, accelerator die, substrate, power delivery, cooling, and assembly yield are one system — a GPU die without matching HBM and advanced-packaging (CoWoS-class) capacity is not a saleable accelerator. This is why HBM supply, not logic wafers, has become the gating constraint on AI accelerator output.\n\nRead HBM through a quant lens rather than a spec-sheet lens: the number that matters is not the headline TB/s but bandwidth per watt, energy per bit moved, and how capacity-per-stack caps the batch and model that fit on one accelerator. On a roofline, HBM sets the memory-bound ceiling — any kernel below the ridge point is limited by these bytes-per-second, not by the GPU's FLOPs.