HBM (High Bandwidth Memory) is specialized 3D-stacked DRAM designed to provide massive memory bandwidth to GPUs and accelerators — achieving 2-5 TB/s bandwidth versus ~100 GB/s for standard DDR, this technology is critical for LLM inference where moving weights from memory to compute is the primary bottleneck.

What Is HBM?

• Definition: 3D-stacked DRAM connected via silicon interposer.
• Innovation: Wide interface (1024+ bits) through vertical stacking.
• Bandwidth: 2-5× higher than any other memory technology.
• Use: AI accelerators (H100, MI300), HPC, graphics.

Why Bandwidth Matters for AI

• Memory-Bound: LLM inference is limited by memory bandwidth, not compute.
• Weight Movement: Every token requires loading all model weights.
• Bottleneck Equation: Tokens/sec ≤ Bandwidth / (2 × Model Size).
• More Bandwidth = More Tokens/Second.

Memory Technology Comparison

Memory    | Bandwidth | Capacity    | Cost     | Use Case
----------|-----------|-------------|----------|------------------
HBM3e     | 4.8 TB/s  | 141 GB      | Very high| H200, MI300X
HBM3      | 3.35 TB/s | 80 GB       | High     | H100
HBM2e     | 2.0 TB/s  | 80 GB       | High     | A100
GDDR6X    | 1.0 TB/s  | 24 GB       | Medium   | RTX 4090
GDDR6     | 0.5 TB/s  | 16-48 GB    | Medium   | RTX 4080
DDR5      | 0.1 TB/s  | 128+ GB     | Low      | CPU RAM

How HBM Works

Architecture:

<svg viewBox="0 0 401 302" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto" role="img"><rect x="0" y="0" width="401" height="302" rx="12" fill="#0d1117"/><g font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,&quot;Liberation Mono&quot;,monospace" font-size="14"><text xml:space="preserve" x="20" y="31.7"><tspan fill="#6e7681">┌─────────────────────────────────────────┐</tspan></text><text xml:space="preserve" x="20" y="50.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">               GPU Die                   </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="69.7"><tspan fill="#6e7681">├─────────────────────────────────────────┤</tspan></text><text xml:space="preserve" x="20" y="88.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">           Silicon Interposer            </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="107.7"><tspan fill="#6e7681">├─────┬─────┬─────┬─────┬─────┬─────┬─────┤</tspan></text><text xml:space="preserve" x="20" y="126.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">HBM  </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">HBM  </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">HBM  </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">HBM  </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">HBM  </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">HBM  </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">HBM  </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="145.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">Stack</tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">Stack</tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">Stack</tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">Stack</tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">Stack</tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">Stack</tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">Stack</tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="164.7"><tspan fill="#6e7681">└─────┴─────┴─────┴─────┴─────┴─────┴─────┘</tspan></text><text xml:space="preserve" x="20" y="183.7"></text><text xml:space="preserve" x="20" y="202.7"><tspan fill="#c9d1d9">Each HBM stack:</tspan></text><text xml:space="preserve" x="20" y="221.7"><tspan fill="#c9d1d9">- 8-12 DRAM dies stacked vertically</tspan></text><text xml:space="preserve" x="20" y="240.7"><tspan fill="#c9d1d9">- Connected via Through-Silicon Vias (TSVs)</tspan></text><text xml:space="preserve" x="20" y="259.7"><tspan fill="#c9d1d9">- 1024-bit wide interface per stack</tspan></text><text xml:space="preserve" x="20" y="278.7"><tspan fill="#c9d1d9">- H100 has 5 stacks = 5120-bit total width</tspan></text></g></svg>

Bandwidth Calculation:

HBM3 (H100):
Width: 5 stacks × 1024 bits = 5120 bits
Speed: 5.2 Gbps per pin
Bandwidth: 5120 × 5.2 Gbps / 8 = 3.35 TB/s

LLM Inference Throughput Limit

Theoretical Maximum:

Max tokens/sec = Memory Bandwidth / Bytes per Token

For 70B model (FP16 = 140 GB):
H100: 3.35 TB/s / 140 GB = 24 tokens/sec (theoretical max)
A100: 2.0 TB/s / 140 GB = 14 tokens/sec
RTX 4090: 1.0 TB/s / 140 GB = 7 tokens/sec

Reality is ~70-80% of theoretical due to overhead

Impact on Different Models:

Model   | Size (FP16) | H100 Max | A100 Max
--------|-------------|----------|----------
7B      | 14 GB       | 239 tok/s| 143 tok/s
13B     | 26 GB       | 129 tok/s| 77 tok/s
70B     | 140 GB      | 24 tok/s | 14 tok/s
405B    | 810 GB      | 4 tok/s* | 2.5 tok/s*
* Multi-GPU required

HBM Generations

Generation | Bandwidth/stack | GPU Example | Year
-----------|-----------------|-------------|------
HBM1       | 128 GB/s        | Fiji        | 2015
HBM2       | 256 GB/s        | V100        | 2016
HBM2e      | 450 GB/s        | A100        | 2020
HBM3       | 665 GB/s        | H100        | 2022
HBM3e      | 1.2 TB/s        | H200        | 2024
HBM4       | 2+ TB/s         | (Future)    | 2025+

Implications for ML

GPU Selection:

• For LLM inference, prioritize bandwidth over FLOPS.
• H100 vs. A100: Only 2× FLOPS but 1.7× bandwidth.
• RTX 4090: Great for small models, limited for 70B+.

Quantization Impact:

Quantization reduces model size → more tokens/sec:

70B model:
FP16 (140 GB): 24 tok/s on H100
INT8 (70 GB):  48 tok/s on H100
INT4 (35 GB):  96 tok/s on H100

4-bit enables ~4× throughput!

Batching Benefit:

Single request: Bandwidth limited
Batching N requests: Same bandwidth reads, N outputs

Batch size 1:  24 tok/s (memory bound)
Batch size 8:  140 tok/s (becoming compute bound)
Batch size 32: 500 tok/s (compute bound)

HBM and memory bandwidth are the physics that govern LLM inference performance — understanding this fundamental constraint explains why quantization, batching, and newer GPUs with more HBM are essential for efficient AI serving.