Hbm4 vs HBM3e

HBM4 vs HBM3E is the generational step that doubles the memory interface rather than speeding it up: HBM3E moves data over a 1024-bit bus at roughly 9.6 Gb/s per pin for about 1.2 TB/s per stack, while HBM4 widens the bus to 2048 bits, pulls pin speed back to around 8 Gb/s, and still reaches roughly 2 TB/s per stack. HBM4 also raises the stack ceiling from 12-Hi to 16-Hi and, for the first time, turns the base die at the bottom of the stack into a foundry-built logic chip. It is the memory generation targeted at NVIDIA Rubin, AMD MI400, and the training clusters that follow them.

Why widen the bus instead of raising the clock. Interface power scales roughly with $C \cdot V^2 \cdot f$ — linearly with frequency and with the square of voltage. Doubling pin speed therefore roughly doubles interface power and makes signal integrity dramatically harder, because every doubling tightens the eye diagram and raises crosstalk across a package already carrying thousands of parallel lines. Doubling the bus width instead buys the same bandwidth at *lower* frequency:

At 1024 bits and 9.6 Gb/s that yields about 1.23 TB/s. At 2048 bits and 8 Gb/s it yields about 2.05 TB/s — a 67% bandwidth gain while each individual wire runs *slower* and more reliably. The energy cost per transferred bit falls even though total throughput rises, which is the entire reason the JEDEC HBM4 specification chose width over speed.

What a 16-Hi stack actually requires. Stacking sixteen DRAM dies inside the same package height budget forces each die to be thinned to roughly 30–40 µm — thinner than a human hair — while remaining flat enough to bond. Three consequences follow:

1. Thermal. The dies farthest from the package lid sit deepest in the thermal stack. The bottom DRAM layers in a 16-Hi stack run hottest, and DRAM retention time halves roughly every 10 °C, so refresh overhead climbs precisely where cooling is worst.
2. Bonding. At 16-Hi, conventional microbump thermocompression bonding approaches its limit. Hybrid bonding — direct copper-to-copper contact with no solder at all — removes the bump gap entirely, shortens the vertical path, and improves heat flow. SK hynix and Samsung both have hybrid-bonded HBM4 paths.
3. Yield. A 16-Hi stack is only as good as its worst die. At 99% per-die yield a 16-high stack yields about 85%; the known-good-die testing burden grows with every layer added.

The base die becomes a logic chip. In HBM3E the base die is a relatively simple DRAM-process interface layer. In HBM4 it becomes a logic die built on a foundry process — TSMC fabricates base dies for SK hynix HBM4 — which lets the customer put memory controller logic, test circuitry, and even light compute directly beneath the DRAM stack. This is the structural reason HBM4 is described as "custom HBM": the base die can differ per customer, so NVIDIA and a hyperscaler can order stacks that are physically different at the bottom while sharing the same DRAM above.

Why AI training is bandwidth-bound, not compute-bound. A modern accelerator can issue far more floating-point operations per second than its memory system can feed. The ratio that matters is arithmetic intensity — FLOPs performed per byte fetched:

Dense matrix multiplication has high arithmetic intensity and keeps the compute units busy. But transformer inference is dominated by attention over the KV cache, which is almost pure streaming: each generated token reads back the entire cache and performs very little arithmetic per byte. That workload sits far to the left of the roofline knee, meaning performance is set almost entirely by memory bandwidth. Doubling HBM bandwidth translates nearly one-for-one into tokens per second in that regime, which is why HBM4 matters more to inference economics than another increment of peak FLOPs would.

Capacity matters as much as speed. A 16-Hi HBM4 stack holds 48–64 GB. Six to eight stacks around a single accelerator therefore put roughly 300–500 GB within one package's reach. The practical effect is how many model parameters and how much KV cache stay resident instead of spilling across the interconnect: every parameter that must be fetched over NVLink or Ethernet instead of HBM costs an order of magnitude more latency and energy. Larger stacks reduce how aggressively a model must be sharded, which in turn reduces the all-reduce traffic that limits scaling efficiency across a cluster.

Who ships it, and when. HBM4 is a three-supplier race with an unusual foundry dependency:

The foundry involvement is the genuinely new structural fact. HBM used to be a pure memory-vendor product; HBM4 makes the bottom die a logic product, which pulls TSMC into the middle of the memory supply chain and gives accelerator designers a place to put their own circuitry inside the memory stack.

What to watch next. Three things determine whether HBM4 delivers its theoretical gain in practice: whether hybrid bonding reaches volume yield at 16-Hi, whether the thermal path through sixteen thinned dies holds refresh overhead down under sustained training load, and whether custom base dies fragment the supply chain into per-customer SKUs that are harder to second-source. Bandwidth on the datasheet is the easy part; the stack has to survive a datacenter duty cycle to be worth the premium.