memory bandwidth high
**HBM (High Bandwidth Memory)** is **specialized 3D-stacked DRAM designed to provide massive memory bandwidth to GPUs and accelerators** — achieving 2-5 TB/s bandwidth versus ~100 GB/s for standard DDR, this technology is critical for LLM inference where moving weights from memory to compute is the primary bottleneck.
**What Is HBM?**
- **Definition**: 3D-stacked DRAM connected via silicon interposer.
- **Innovation**: Wide interface (1024+ bits) through vertical stacking.
- **Bandwidth**: 2-5× higher than any other memory technology.
- **Use**: AI accelerators (H100, MI300), HPC, graphics.
**Why Bandwidth Matters for AI**
- **Memory-Bound**: LLM inference is limited by memory bandwidth, not compute.
- **Weight Movement**: Every token requires loading all model weights.
- **Bottleneck Equation**: Tokens/sec ≤ Bandwidth / (2 × Model Size).
- **More Bandwidth = More Tokens/Second**.
**Memory Technology Comparison**
```
Memory | Bandwidth | Capacity | Cost | Use Case
----------|-----------|-------------|----------|------------------
HBM3e | 4.8 TB/s | 141 GB | Very high| H200, MI300X
HBM3 | 3.35 TB/s | 80 GB | High | H100
HBM2e | 2.0 TB/s | 80 GB | High | A100
GDDR6X | 1.0 TB/s | 24 GB | Medium | RTX 4090
GDDR6 | 0.5 TB/s | 16-48 GB | Medium | RTX 4080
DDR5 | 0.1 TB/s | 128+ GB | Low | CPU RAM
```
**How HBM Works**
**Architecture**:
```svg
```
**Bandwidth Calculation**:
```
HBM3 (H100):
Width: 5 stacks × 1024 bits = 5120 bits
Speed: 5.2 Gbps per pin
Bandwidth: 5120 × 5.2 Gbps / 8 = 3.35 TB/s
```
**LLM Inference Throughput Limit**
**Theoretical Maximum**:
```
Max tokens/sec = Memory Bandwidth / Bytes per Token
For 70B model (FP16 = 140 GB):
H100: 3.35 TB/s / 140 GB = 24 tokens/sec (theoretical max)
A100: 2.0 TB/s / 140 GB = 14 tokens/sec
RTX 4090: 1.0 TB/s / 140 GB = 7 tokens/sec
Reality is ~70-80% of theoretical due to overhead
```
**Impact on Different Models**:
```
Model | Size (FP16) | H100 Max | A100 Max
--------|-------------|----------|----------
7B | 14 GB | 239 tok/s| 143 tok/s
13B | 26 GB | 129 tok/s| 77 tok/s
70B | 140 GB | 24 tok/s | 14 tok/s
405B | 810 GB | 4 tok/s* | 2.5 tok/s*
* Multi-GPU required
```
**HBM Generations**
```
Generation | Bandwidth/stack | GPU Example | Year
-----------|-----------------|-------------|------
HBM1 | 128 GB/s | Fiji | 2015
HBM2 | 256 GB/s | V100 | 2016
HBM2e | 450 GB/s | A100 | 2020
HBM3 | 665 GB/s | H100 | 2022
HBM3e | 1.2 TB/s | H200 | 2024
HBM4 | 2+ TB/s | (Future) | 2025+
```
**Implications for ML**
**GPU Selection**:
- For LLM inference, prioritize bandwidth over FLOPS.
- H100 vs. A100: Only 2× FLOPS but 1.7× bandwidth.
- RTX 4090: Great for small models, limited for 70B+.
**Quantization Impact**:
```
Quantization reduces model size → more tokens/sec:
70B model:
FP16 (140 GB): 24 tok/s on H100
INT8 (70 GB): 48 tok/s on H100
INT4 (35 GB): 96 tok/s on H100
4-bit enables ~4× throughput!
```
**Batching Benefit**:
```
Single request: Bandwidth limited
Batching N requests: Same bandwidth reads, N outputs
Batch size 1: 24 tok/s (memory bound)
Batch size 8: 140 tok/s (becoming compute bound)
Batch size 32: 500 tok/s (compute bound)
```
HBM and memory bandwidth are **the physics that govern LLM inference performance** — understanding this fundamental constraint explains why quantization, batching, and newer GPUs with more HBM are essential for efficient AI serving.