compute bound vs memory bound

Compute bound vs. memory bound describes whether a GPU workload's performance is limited by arithmetic computation speed (FLOPS) or by the rate of reading/writing data from memory (bandwidth), determining which optimization strategies are effective. Compute bound: operation performs many arithmetic operations per byte of data loaded—limited by GPU FLOPS, not memory bandwidth. Examples: large matrix multiplications (GEMM), convolutions with high arithmetic intensity. Characterized by high GPU compute utilization, adding more computation doesn't help but faster hardware does. Memory bound: operation performs few computations per byte loaded—limited by memory bandwidth, GPU compute units idle waiting for data. Examples: element-wise operations (activation, normalization), attention score computation, autoregressive decoding with small batch size. Arithmetic intensity: the ratio of compute operations to memory operations (FLOPS/byte). The "roofline model" plots achievable performance against arithmetic intensity: below the ridge point (intersection), workload is memory-bound; above, compute-bound. LLM inference phases: (1) Prefill—processing input tokens, large batch matrix multiplications → compute-bound; (2) Decode—generating tokens one at a time, reading all weights for single token → memory-bound (the bottleneck). H100 GPU balance point: 989 TFLOPS (FP16) / 3.35 TB/s (HBM3) = ~295 ops/byte. Operations with arithmetic intensity below 295 are memory-bound. Optimization by regime: (1) Memory-bound—batch more requests (increase arithmetic intensity), quantize weights (reduce bytes), use faster memory, kernel fusion (reduce memory trips); (2) Compute-bound—use lower precision (FP16→FP8→INT8), sparse computation, efficient algorithms, faster hardware. This distinction is fundamental to choosing the right optimization strategy for any GPU workload in deep learning.