memory bandwidth

Memory bandwidth—the rate of data transfer between processor and memory—is often the primary bottleneck limiting AI inference performance. Modern accelerators achieve hundreds of TFLOPS of compute capacity but are frequently starved for data. Memory bandwidth is measured in GB/s or TB/s: consumer GPUs provide 500-1000 GB/s, while data center accelerators with HBM achieve 2-3 TB/s. For LLM inference, bandwidth requirements are dominated by model weight loading: generating one token requires reading all parameters once (batch=1), meaning a 70B parameter model in FP16 needs 140GB read per token. At 2TB/s bandwidth, this limits throughput to ~14 tokens/second regardless of compute capability. Techniques to mitigate bandwidth constraints include: quantization (INT8/INT4 reduces bytes per parameter 2-4x), batching (amortizes weight loading across multiple sequences), speculative decoding (generates multiple tokens per weight load), and KV cache optimization (reduces non-weight memory traffic). System design must balance bandwidth, compute, and memory capacity. The emergence of bandwidth as the key bottleneck drives chip architecture toward higher HBM stacks, Processing-in-Memory (PIM), and on-chip SRAM expansion.