memory bound
**Memory bound describes a workload whose elapsed time is limited primarily by data supply rather than arithmetic capability.** It explains why enormous advertised FLOPS can remain idle during LLM decode, embeddings, elementwise transforms, reductions, sparse operations, and irregular analytics. In the roofline model, arithmetic intensity below machine balance places a kernel on the rising bandwidth roof, where attainable operations per second equal intensity times sustained bytes per second. A professional performance claim defines workload, useful work, input and output shapes, numerical format, batch and concurrency, warmup and measurement interval, hardware and software versions, power state, correctness tolerance, and aggregation method. Peak specifications are ceilings under particular conditions; delivered behavior includes utilization, data movement, synchronization, control overhead, and tail effects. The boundary must name the memory level: a kernel can be HBM bound, cache bound, shared-memory bank bound, host-memory bound, or interconnect bound.
**Architecture, quantitative model, and operating behavior.** Low-reuse operations fetch many bytes for little arithmetic. Single-token autoregressive decode reads large weight matrices for a small amount of output; batching reuses weights across more tokens and raises intensity. Fusion avoids intermediate writes and moves the point rightward. Caches, prefetch, coalescing, tiling, compression, quantization, sparsity, batching, and recomputation trade capacity, compute, or latency for fewer external bytes. Higher-bandwidth HBM raises the diagonal roof but does not correct random latency or poor locality by itself. Bandwidth-bound streaming, latency-bound pointer chasing, capacity-bound paging, cache-thrashing, TLB-bound, and network-bound regimes may all starve compute but require different evidence and solutions. Useful analysis separates arithmetic, memory hierarchy, interconnect, storage, control, and queuing. It counts operations and bytes at each boundary, identifies dependencies and reuse, estimates ideal ceilings, and then uses counters and traces to explain the gap between the model and measurement. Ratios without a clearly named numerator and denominator invite invalid comparisons. Report useful throughput together with latency distribution, utilization, arithmetic intensity, achieved bandwidth, cache hit rate, occupancy, communication time, memory capacity, power, energy per result, quality, and cost. Include median and tail behavior, sustained rather than burst operation, repeated trials, and uncertainty. A faster approximation is not equivalent unless it meets the same accuracy and service constraints.
**Implementation, hardware mapping, and bottlenecks.** Count mandatory and avoidable bytes, fuse elementwise chains, retain tiles locally, quantize weights and KV cache, batch compatible requests, reorganize layouts, prefetch predictably, and shard so traffic follows topology. HBM stacks and memory controllers help streaming demand; larger caches help reuse; more channels help parallelism; on-chip SRAM enables explicit tiling. Narrow links or package power can become the true memory ceiling. Calling every low-utilization kernel memory bound, optimizing FLOPS, compressing data with excessive decode cost, batching beyond latency SLO, or ignoring write amplification and cache misses leads to wrong action. Begin with a correct reference and representative shapes. Profile end to end, classify the dominant resource, inspect kernel and system timelines, change one bottleneck at a time, and remeasure because optimization moves pressure elsewhere. Tiling, fusion, batching, vectorization, layout, precision, compression, overlap, prefetch, sharding, and algorithm choice are useful only when they reduce the limiting resource. The execution path spans registers, local SRAM and caches, HBM or GDDR, host DRAM, PCIe or coherent links, scale-up fabric, network, and storage. Compute units consume tensors only when compilers and kernels issue enough independent work and the hierarchy supplies operands. Package wiring, memory stacks, clocks, voltage, thermal headroom, and power delivery determine sustained limits. Frequent mistakes include quoting peak instead of achieved rates, omitting data conversion and transfer, measuring a cached toy input, timing asynchronous work without synchronization, mixing decimal and binary units, ignoring warmup or throttling, changing precision or quality, averaging away tails, and optimizing a component that is not on the critical path.
**Measurement, validation, and engineering controls.** Measure bytes at cache/HBM/interconnect levels, achieved bandwidth, stall reasons, latency, cache and TLB hit rates, and sensitivity to bandwidth, clocks, batch, compression, and reuse. Bytes per useful result, arithmetic intensity, achieved/peak bandwidth, compute idle fraction, cache/TLB behavior, batch, latency, throughput, energy, and quality matter. A byte accounting model reconciled with counters separates mandatory model reads from spills, layout copies, allocator traffic, and redundant intermediates. Verification combines analytical bounds, microbenchmarks, hardware counters, kernel timelines, end-to-end traces, scaling sweeps, sensitivity to batch and shape, cold and warm runs, long-duration thermal tests, correctness comparisons, fault and congestion tests, and independent reproduction. Roofline and queueing models guide diagnosis but must be calibrated against the deployed machine. Benchmark code, datasets, model and compiler artifacts, drivers, firmware, topology, clock and power settings, environment, commands, raw samples, counter traces, and analysis notebooks remain versioned. Continuous tests detect regressions in quality, latency, throughput, bandwidth, memory, power, and cost, with thresholds chosen from variance rather than a single run. Published comparisons disclose configuration, exclusions, tuning effort, measurement boundary, quality criteria, and uncertainty. Energy and carbon claims distinguish chip, IT, and facility boundaries and avoid extrapolating one benchmark to all workloads. Owners review regressions and retain evidence sufficient to reproduce decisions.
| Workload/operation | Typical intensity | Likely limit | Primary lever | Verification |
|---|---|---|---|---|
| Large batched GEMM | High | Compute throughput | Tensor cores/precision/tiles | Roofline and instruction rate |
| LLM single-token decode | Low | Weight-memory bandwidth | Batch/compress/cache | Bytes per token |
| Elementwise chain | Low unless fused | Memory traffic/launch | Fusion/vectorization | Kernel and byte count |
| Large convolution | Moderate to high | Shape dependent | Tiling/algorithm | Sweep batch/channels |
| Embedding lookup | Very low/irregular | Memory latency/bandwidth | Layout/cache/shard | Hit rate and stalls |
| Collective communication | Network intensity | Fabric/latency | Topology/overlap/compress | Link counters/timeline |
```svg
```
**Selection and system-level application.** First reduce bytes and improve reuse, then add bandwidth; accept batching or compression only within latency and quality constraints. LLM token decode, embedding lookup, graph traversal, reductions, vector operations, sparse kernels, and streaming transforms are frequently memory bound. Memory limitation is co-designed with model architecture, precision, cache, batching, allocator, package, interconnect, serving policy, and power. Optimization is a system exercise across algorithms, precision, kernels, compiler, runtime, accelerator, memory, interconnect, scheduler, serving policy, cooling, and facility limits. Removing one ceiling often exposes another, so architecture decisions should optimize time and energy to a useful result rather than an isolated metric. A professional performance claim defines workload, useful work, input and output shapes, numerical format, batch and concurrency, warmup and measurement interval, hardware and software versions, power state, correctness tolerance, and aggregation method. Peak specifications are ceilings under particular conditions; delivered behavior includes utilization, data movement, synchronization, control overhead, and tail effects. Report useful throughput together with latency distribution, utilization, arithmetic intensity, achieved bandwidth, cache hit rate, occupancy, communication time, memory capacity, power, energy per result, quality, and cost. Include median and tail behavior, sustained rather than burst operation, repeated trials, and uncertainty. A faster approximation is not equivalent unless it meets the same accuracy and service constraints. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.