memory bound

**Memory-bound vs compute-bound** describes whether a **workload is limited by memory bandwidth or computational throughput** — understanding this bottleneck is critical for optimizing neural network inference and training performance. **What Is Memory-Bound vs Compute-Bound?** - **Memory-Bound**: Limited by how fast data moves to/from memory. - **Compute-Bound**: Limited by how fast calculations are performed. - **Diagnosis**: Profile to find which saturates first. - **Optimization**: Different strategies for each bottleneck. - **Examples**: Attention is memory-bound, convolutions often compute-bound. **Why This Distinction Matters** - **Optimization Strategy**: Wrong focus wastes effort. - **Hardware Selection**: Memory-bound → faster memory; compute-bound → more cores. - **Batching**: Helps compute-bound more than memory-bound. - **Quantization**: Helps memory-bound through smaller data transfers. - **Architecture Design**: Informs model choices (attention vs conv). **Identifying the Bottleneck** **Memory-Bound Indicators**: - Low GPU utilization despite full memory bandwidth. - Small batch sizes, large models. - Element-wise operations, attention mechanisms. **Compute-Bound Indicators**: - High GPU utilization, memory bandwidth available. - Large batch sizes, matrix multiplications. - Convolutions, dense layers. **Optimization Strategies** **Memory-Bound**: Quantization, operator fusion, Flash Attention, smaller models. **Compute-Bound**: Larger batches, tensor cores, mixed precision, more compute. Understanding bottlenecks enables **targeted optimization** — fix the actual limiting factor.