Memory bandwidth is the sustained rate at which useful bytes move between a processor and a memory level. It sets the performance ceiling for workloads whose arithmetic units cannot reuse data enough to hide operand traffic, creating the memory wall in AI and HPC. Peak bandwidth follows interface width, transfers per clock, channels, stacks, and frequency; achieved bandwidth is lower because of protocol, access pattern, refresh, contention, reads versus writes, and controller efficiency. A professional performance claim defines workload, useful work, input and output shapes, numerical format, batch and concurrency, warmup and measurement interval, hardware and software versions, power state, correctness tolerance, and aggregation method. Peak specifications are ceilings under particular conditions; delivered behavior includes utilization, data movement, synchronization, control overhead, and tail effects. Always name source and destination, read/write mix, payload versus wire bytes, decimal or binary units, locality, concurrency, access size, and sustained interval.

Architecture, quantitative model, and operating behavior. On-chip register files and SRAM offer enormous aggregate local bandwidth at small capacity. HBM3E-class devices deliver several TB/s through wide package interfaces, high-end GDDR systems around the TB/s class, DDR5 channels tens of GB/s each, and LPDDR prioritizes energy. Controllers schedule banks and channels, caches filter traffic, coalescers combine requests, prefetchers anticipate streams, and tiling reuses loaded data. Little’s-law reasoning links latency, concurrency, and throughput: enough requests must remain in flight to fill a wide interface. Read, write, bidirectional aggregate, random, sequential, copy, triad, cache, local SRAM, HBM, GDDR, DDR, LPDDR, and effective application bandwidth measure different things and should not be interchanged. Useful analysis separates arithmetic, memory hierarchy, interconnect, storage, control, and queuing. It counts operations and bytes at each boundary, identifies dependencies and reuse, estimates ideal ceilings, and then uses counters and traces to explain the gap between the model and measurement. Ratios without a clearly named numerator and denominator invite invalid comparisons. Report useful throughput together with latency distribution, utilization, arithmetic intensity, achieved bandwidth, cache hit rate, occupancy, communication time, memory capacity, power, energy per result, quality, and cost. Include median and tail behavior, sustained rather than burst operation, repeated trials, and uncertainty. A faster approximation is not equivalent unless it meets the same accuracy and service constraints.

Implementation, hardware mapping, and bottlenecks. Increase locality and arithmetic intensity through tiling and fusion, align and coalesce accesses, use compact formats, avoid redundant materialization, batch for reuse, overlap copies, distribute traffic across channels, and compress when decode cost is cheaper than bytes saved. HBM stacks, PHYs, package routing, memory controllers, banks, refresh, ECC, cache slices, NoC, and power budget set the ceiling. Thermal and simultaneous compute activity can lower the sustainable rate. STREAM-like peaks do not predict gathers, writes, or cache-conflicted kernels; counting requested instead of transferred bytes hides amplification; stating SRAM as unlimited ignores ports and banks; adding bandwidth cannot fix latency-limited dependencies. Begin with a correct reference and representative shapes. Profile end to end, classify the dominant resource, inspect kernel and system timelines, change one bottleneck at a time, and remeasure because optimization moves pressure elsewhere. Tiling, fusion, batching, vectorization, layout, precision, compression, overlap, prefetch, sharding, and algorithm choice are useful only when they reduce the limiting resource. The execution path spans registers, local SRAM and caches, HBM or GDDR, host DRAM, PCIe or coherent links, scale-up fabric, network, and storage. Compute units consume tensors only when compilers and kernels issue enough independent work and the hierarchy supplies operands. Package wiring, memory stacks, clocks, voltage, thermal headroom, and power delivery determine sustained limits. Frequent mistakes include quoting peak instead of achieved rates, omitting data conversion and transfer, measuring a cached toy input, timing asynchronous work without synchronization, mixing decimal and binary units, ignoring warmup or throttling, changing precision or quality, averaging away tails, and optimizing a component that is not on the critical path.

Measurement, validation, and engineering controls. Benchmark multiple access patterns and sizes, use counters for bytes at every level, measure achieved fraction of peak, vary concurrency and alignment, test NUMA/channel placement, and run sustained mixed workloads. GB/s or TB/s, utilization, latency, transactions, cache hit, row-buffer hit, bank conflicts, amplification, concurrency, energy per byte, and application throughput matter. Stride sweeps, cache-size sweeps, read/write mixes, and counter reconciliation reveal whether the limit is interface bandwidth, locality, latency, or instruction issue. Verification combines analytical bounds, microbenchmarks, hardware counters, kernel timelines, end-to-end traces, scaling sweeps, sensitivity to batch and shape, cold and warm runs, long-duration thermal tests, correctness comparisons, fault and congestion tests, and independent reproduction. Roofline and queueing models guide diagnosis but must be calibrated against the deployed machine. Benchmark code, datasets, model and compiler artifacts, drivers, firmware, topology, clock and power settings, environment, commands, raw samples, counter traces, and analysis notebooks remain versioned. Continuous tests detect regressions in quality, latency, throughput, bandwidth, memory, power, and cost, with thresholds chosen from variance rather than a single run. Published comparisons disclose configuration, exclusions, tuning effort, measurement boundary, quality criteria, and uncertainty. Energy and carbon claims distinguish chip, IT, and facility boundaries and avoid extrapolating one benchmark to all workloads. Owners review regressions and retain evidence sufficient to reproduce decisions.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Memory Bandwidth — The Real AI Bottleneck</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">LLM decode is memory-bound: every token loads all weights from HBM — bandwidth, not FLOPs, sets tokens/second</text>

  <!-- === TOP: Memory hierarchy with bandwidth === -->
  <rect x="25" y="62" width="710" height="130" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="80" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Memory Hierarchy — Bandwidth at Each Level</text>

  <!-- Registers/SRAM -->
  <rect x="45" y="92" width="110" height="40" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="100" y="110" fill="#6ee7b7" font-size="8.5" text-anchor="middle" font-weight="600">Registers / SRAM</text>
  <text x="100" y="124" fill="#34d399" font-size="8" text-anchor="middle">~100 TB/s, 1 cycle</text>

  <!-- L2 cache -->
  <rect x="170" y="92" width="100" height="40" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.9"/>
  <text x="220" y="110" fill="#93c5fd" font-size="8.5" text-anchor="middle" font-weight="600">L2 Cache</text>
  <text x="220" y="124" fill="#60a5fa" font-size="8" text-anchor="middle">~10 TB/s, 10 cycles</text>

  <!-- HBM -->
  <rect x="285" y="88" width="130" height="48" rx="4" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.1"/>
  <text x="350" y="108" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">HBM3e (GPU)</text>
  <text x="350" y="122" fill="#fbbf24" font-size="8.5" text-anchor="middle">3.35 TB/s (H100)</text>
  <text x="350" y="134" fill="#6b7684" font-size="7" text-anchor="middle">80 GB, ~100 cycles</text>

  <!-- GDDR -->
  <rect x="430" y="92" width="100" height="40" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="0.9"/>
  <text x="480" y="110" fill="#c4b5fd" font-size="8.5" text-anchor="middle" font-weight="600">GDDR6X</text>
  <text x="480" y="124" fill="#c4b5fd" font-size="8" text-anchor="middle">1 TB/s (RTX 4090)</text>

  <!-- DDR5 -->
  <rect x="545" y="92" width="90" height="40" rx="3" fill="#0b1220" stroke="#8b98a5" stroke-width="0.8"/>
  <text x="590" y="110" fill="#8b98a5" font-size="8.5" text-anchor="middle" font-weight="600">DDR5</text>
  <text x="590" y="124" fill="#8b98a5" font-size="8" text-anchor="middle">90 GB/s (CPU)</text>

  <!-- NVLink/PCIe -->
  <rect x="650" y="92" width="70" height="40" rx="3" fill="#1a0f0f" stroke="#f87171" stroke-width="0.8"/>
  <text x="685" y="110" fill="#f87171" font-size="8" text-anchor="middle" font-weight="600">Network</text>
  <text x="685" y="124" fill="#f87171" font-size="7.5" text-anchor="middle">NVLink 900 GB/s</text>

  <!-- Bandwidth wall annotation -->
  <text x="380" y="155" fill="#f87171" font-size="9" text-anchor="middle" font-weight="600">The bandwidth wall: compute grows 2× per year, bandwidth grows ~1.3× → the gap widens every generation</text>
  <text x="380" y="172" fill="#6b7684" font-size="8" text-anchor="middle">H100: 3958 TFLOPS (FP8) but only 3.35 TB/s HBM → arithmetic intensity must be &gt; 1181 ops/byte to be compute-bound</text>

  <!-- === MIDDLE LEFT: Why bandwidth limits AI === -->
  <rect x="25" y="200" width="350" height="115" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="218" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Why LLM Inference Is Bandwidth-Bound</text>

  <text x="45" y="240" fill="#f87171" font-size="8.5" font-weight="600">Decode phase (autoregressive):</text>
  <text x="45" y="256" fill="#8b98a5" font-size="8.5">1 token → load ALL weights from HBM (70B = 140 GB)</text>
  <text x="45" y="274" fill="#fbbf24" font-size="8.5" font-weight="600">Arithmetic intensity:</text>
  <text x="45" y="290" fill="#8b98a5" font-size="8.5">batch=1: 1 FLOP per byte loaded (AI = 2 for matmul)</text>
  <text x="45" y="290" fill="#8b98a5" font-size="8.5">batch=1: ~2 FLOPs/byte — far below roofline</text>
  <text x="45" y="306" fill="#34d399" font-size="8.5">Larger batch → higher AI → closer to compute-bound</text>

  <!-- === MIDDLE RIGHT: Solutions === -->
  <rect x="390" y="200" width="345" height="115" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="218" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Maximizing Effective Bandwidth</text>

  <text x="410" y="240" fill="#34d399" font-size="8.5" font-weight="600">Quantization (INT4/FP8):</text>
  <text x="560" y="240" fill="#8b98a5" font-size="8.5">4× less bytes to load per weight</text>
  <text x="410" y="258" fill="#60a5fa" font-size="8.5" font-weight="600">Batching:</text>
  <text x="470" y="258" fill="#8b98a5" font-size="8.5">amortize weight loads across requests</text>
  <text x="410" y="276" fill="#fbbf24" font-size="8.5" font-weight="600">GQA/MQA:</text>
  <text x="475" y="276" fill="#8b98a5" font-size="8.5">fewer KV heads = less KV cache traffic</text>
  <text x="410" y="294" fill="#c4b5fd" font-size="8.5" font-weight="600">FlashDecoding:</text>
  <text x="505" y="294" fill="#8b98a5" font-size="8.5">split-K for better parallelism</text>
  <text x="410" y="312" fill="#f87171" font-size="8.5" font-weight="600">More HBM stacks:</text>
  <text x="530" y="312" fill="#8b98a5" font-size="8.5">H200: 4.8 TB/s, B200: 8 TB/s</text>

  <!-- === BOTTOM: Hardware comparison === -->
  <rect x="25" y="325" width="710" height="78" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="343" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Bandwidth by Hardware (2024-2025)</text>

  <text x="95" y="367" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">H100 SXM</text>
  <text x="95" y="381" fill="#8b98a5" font-size="8" text-anchor="middle">3.35 TB/s (HBM3)</text>
  <text x="95" y="393" fill="#6b7684" font-size="7.5" text-anchor="middle">80 GB, 5 stacks</text>

  <text x="240" y="367" fill="#34d399" font-size="9" text-anchor="middle" font-weight="600">H200</text>
  <text x="240" y="381" fill="#8b98a5" font-size="8" text-anchor="middle">4.8 TB/s (HBM3e)</text>
  <text x="240" y="393" fill="#6b7684" font-size="7.5" text-anchor="middle">141 GB, 6 stacks</text>

  <text x="380" y="367" fill="#60a5fa" font-size="9" text-anchor="middle" font-weight="600">B200</text>
  <text x="380" y="381" fill="#8b98a5" font-size="8" text-anchor="middle">8 TB/s (HBM3e)</text>
  <text x="380" y="393" fill="#6b7684" font-size="7.5" text-anchor="middle">192 GB, 8 stacks</text>

  <text x="520" y="367" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">MI300X (AMD)</text>
  <text x="520" y="381" fill="#8b98a5" font-size="8" text-anchor="middle">5.3 TB/s (HBM3)</text>
  <text x="520" y="393" fill="#6b7684" font-size="7.5" text-anchor="middle">192 GB, 8 stacks</text>

  <text x="665" y="367" fill="#f87171" font-size="9" text-anchor="middle" font-weight="600">Groq LPU</text>
  <text x="665" y="381" fill="#8b98a5" font-size="8" text-anchor="middle">80 TB/s (SRAM only)</text>
  <text x="665" y="393" fill="#6b7684" font-size="7.5" text-anchor="middle">230 MB, no HBM</text>

  <!-- Key insight -->
  <rect x="25" y="411" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="426" fill="#fbbf24" font-size="9" text-anchor="middle">LLM token rate ≈ HBM bandwidth ÷ model size: 3.35 TB/s ÷ 140 GB (70B FP16) = 24 tokens/s per user (theory max).</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Memory bandwidth is the currency of AI inference: whoever moves bytes fastest generates tokens fastest.</text>
</svg>

Selection and system-level application. Choose HBM for maximum accelerator bandwidth, GDDR for board-level balance, DDR5 for host capacity and channels, LPDDR for energy-sensitive systems, and SRAM for explicitly managed local reuse. LLM decode, embeddings, graph analytics, stencils, sparse operations, video, databases, and memory copies are bandwidth-sensitive. Bandwidth interacts with capacity, latency, compute balance, cache, model precision, batching, package, NUMA, interconnect, power, and thermal design. Optimization is a system exercise across algorithms, precision, kernels, compiler, runtime, accelerator, memory, interconnect, scheduler, serving policy, cooling, and facility limits. Removing one ceiling often exposes another, so architecture decisions should optimize time and energy to a useful result rather than an isolated metric. A professional performance claim defines workload, useful work, input and output shapes, numerical format, batch and concurrency, warmup and measurement interval, hardware and software versions, power state, correctness tolerance, and aggregation method. Peak specifications are ceilings under particular conditions; delivered behavior includes utilization, data movement, synchronization, control overhead, and tail effects. Report useful throughput together with latency distribution, utilization, arithmetic intensity, achieved bandwidth, cache hit rate, occupancy, communication time, memory capacity, power, energy per result, quality, and cost. Include median and tail behavior, sustained rather than burst operation, repeated trials, and uncertainty. A faster approximation is not equivalent unless it meets the same accuracy and service constraints. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.