Memory Bandwidth Optimization is the performance engineering discipline of maximizing the effective utilization of available memory bandwidth in compute kernels — the critical challenge for bandwidth-bound applications where the GPU or CPU is waiting for data from DRAM rather than executing compute instructions. Most deep learning inference workloads, large language model generation (decode phase), sparse computations, and data-processing kernels are memory bandwidth bound rather than compute bound, making memory access optimization the primary path to performance improvement.
Bandwidth Bound vs. Compute Bound
- Roofline Model: Performance = min(Peak FLOPS, Arithmetic Intensity × Memory Bandwidth).
- Arithmetic Intensity (AI): FLOPs per byte of data loaded from memory.
- Memory bound: AI < AI_ridge_point → limited by bandwidth, not compute.
- Compute bound: AI > AI_ridge_point → limited by peak FLOPS.
LLM Decode is Memory Bandwidth Bound
- During token generation (autoregressive decode): Load all model weights (7B × 2 bytes = 14 GB for FP16 7B model) to generate ONE token.
- Arithmetic intensity: ~1 FLOP per byte → extremely memory bound.
- A100 GPU: 2 TB/s bandwidth → generates ~140 billion parameters worth of tokens/second → ~10 tokens/second for 7B model (with batching).
- Batching: Batch 100 requests simultaneously → same 14 GB loaded → 100× more compute reuse → approaches compute bound.
Memory Hierarchy and Effective Bandwidth
| Level | Bandwidth (A100) | Latency | Reuse Factor |
|-------|-----------------|---------|-------------|
| Registers | >80 TB/s | 1 cycle | Per-thread |
| L1/Shared | 19 TB/s | 20 cycles | Per-CTA |
| L2 | 4 TB/s | 200 cycles | Per-GPU |
| HBM (DRAM) | 2 TB/s | 600 cycles | Global |
| PCIe (host) | 64 GB/s | µs | Host |
Techniques to Improve Memory Bandwidth Utilization
1. Coalesced Memory Access
- All threads in a warp must access contiguous, aligned memory addresses.
- Non-coalesced: 32 threads × random addresses → 32 separate DRAM transactions → 32× bandwidth waste.
- Coalesced: 32 threads × consecutive addresses → 1 DRAM transaction → full bandwidth utilized.
2. Shared Memory Tiling
- Load tile of input data from global memory → shared memory → compute from shared memory.
- Amortize global memory load over multiple compute operations → increase arithmetic intensity.
- Shared memory bandwidth: ~10× DRAM bandwidth → huge speedup for reused data.
3. Fused Kernels
- Instead of: Load data → compute → store → load → compute → store (multiple global memory round-trips).
- Fused: Load once → compute everything → store once → reduce global memory traffic.
- Example: Fused LayerNorm + attention: Single kernel pass through activations → 3× less bandwidth.
4. Quantization for Bandwidth Reduction
- FP16 → INT8: 2× less data → 2× more weights per second through bandwidth.
- INT4 (4-bit): 4× less data vs. FP16 → 4× bandwidth improvement for weight loading.
- Activation quantization: Input activations also smaller → further bandwidth reduction.
5. KV Cache Compression
- LLM inference KV cache grows linearly with sequence length → bandwidth bound.
- Group Query Attention (GQA): Share KV heads across query groups → reduce KV cache size 4–8×.
- Paged attention: Virtual memory for KV cache → reduces memory waste → better batching.
6. Memory Layout Optimization
- Row-major vs. column-major: Must match access pattern to avoid strided access.
- Structure-of-Arrays (SoA) vs. Array-of-Structures (AoS): SoA enables coalesced access.
- Channel-last format for convolution: NHWC (batch, height, width, channel) → coalesced channel access.
7. Prefetching
- Instruction-level prefetch: Tell memory controller to load next data before it is needed.
- Software prefetch: Initiate async memory copy (cudaMemcpyAsync) while computing current batch.
- Hardware prefetch: GPU L2 prefetcher predicts sequential access patterns → automatic.
Tools for Memory Bandwidth Analysis
- Nsight Compute: Per-kernel memory throughput, DRAM utilization, L1/L2 hit rate.
- Roofline chart: Plot actual kernel on roofline → determine if memory or compute bound.
- DRAM bandwidth utilization metric: Actual vs. peak HBM bandwidth (target >70% for memory-bound kernels).
Memory bandwidth optimization is the essential performance discipline for the inference era of AI — as language models with billions to hundreds of billions of parameters are deployed for real-time inference, the rate at which model weights can be streamed from memory to compute units determines user-experienced latency, server throughput, and ultimately the economics of AI service delivery, making bandwidth-aware kernel design one of the highest-value skills in modern systems programming.