GPU computing uses graphics processors for general-purpose parallel work beyond raster graphics. GPUs devote large area to throughput-oriented arithmetic and supply high memory bandwidth, while hardware scheduling swaps among ready warps or wavefronts to hide latency. This organization fits dense linear algebra, simulation, image/video, analytics, and many AI workloads. Thousands of advertised cores do not behave like thousands of independent CPU cores. SIMT or SIMD groups share instruction issue; branch divergence wastes lanes; global memory latency is hidden only with independent work; cache and local memory reward reuse; kernel launches and transfers impose boundaries. Programming ecosystems include CUDA on NVIDIA, ROCm and HIP on AMD, oneAPI and SYCL in Intel and cross-vendor contexts, OpenCL, Vulkan compute, and vendor libraries. The best path depends on hardware, software, portability, and workload. A production specification starts with workloads and user-visible objectives rather than API names or peak throughput. It records input sizes and distributions, arithmetic precision, control divergence, locality, working-set size, transfer volume, synchronization, latency percentiles, throughput, power, thermal limits, device and driver versions, compiler flags, and correctness tolerance. Measurements identify hardware, software, clocks, power mode, warmup, repetitions, and whether results are theoretical, simulated, or observed. A benchmark without this context cannot guide architecture or purchasing.

Execution model, software stack, and data movement. The host prepares data and command streams, a runtime dispatches grids or work-groups, GPU front ends distribute groups to compute units, lanes execute vector/SIMT instructions, caches and high-bandwidth memory feed operands, and synchronization exposes results. The complete execution stack includes application or model code, a framework or graphics engine, graph capture or shader compilation, intermediate representations, optimization and scheduling, a runtime API, user-mode and kernel drivers, command queues, device firmware, GPU or accelerator hardware, memory, and synchronization with the host and peer devices. Performance can be lost at any boundary through graph breaks, state changes, tiny launches, allocation, copies, serialization, cache misses, occupancy limits, or unsupported fallback. Treating one kernel as the system hides the cost that users experience. Optimization is a sequence of evidence-based transformations: establish correctness and a baseline, profile representative inputs, classify compute, memory, latency, launch, and synchronization limits, improve algorithms and data layout, fuse compatible work, tile for locality, vectorize or map to SIMT, overlap transfers and execution, tune launch geometry, reduce precision only with accuracy checks, and retest the complete workload. Higher occupancy is not automatically faster; register pressure, shared memory, instruction mix, cache behavior, and memory-level parallelism must be interpreted together.

GPU implementation requires evidence-based performance engineering. Characterize parallelism and locality, use tuned libraries for standard math, partition large work, make accesses contiguous, tile reusable data, reduce host-device boundaries, use mixed precision safely, overlap independent work, and profile before writing architecture-specific code. Implementation links software abstractions to finite hardware resources. Teams define ownership and lifetime of buffers, explicit dependencies, queue and stream policy, command reuse, descriptor or argument binding, memory placement, alignment, batching, error propagation, timeout and recovery, telemetry, and deterministic build artifacts. Hardware-aware code remains parameterized by capability queries instead of assuming one device generation. Libraries are preferred for mature primitives, while custom kernels are justified by workload shape, fusion opportunity, or missing functionality. Useful models separate host time, queueing, transfer, kernel, synchronization, and presentation or network time. Roofline analysis relates arithmetic intensity to compute and memory ceilings; queuing models expose concurrency and tail latency; trace-driven and cycle models reveal contention; counters attribute stalls and cache behavior. Models are calibrated against progressively more detailed evidence and include uncertainty. The goal is not one exact prediction but a decision: which bottleneck matters, which design is Pareto-efficient, and what measurement would reduce risk.

Verification, portability, and production controls. Check CPU/reference agreement, precision and reduction order, races, irregular sizes, multi-GPU communication, thermal throttling, memory exhaustion, kernel timeout, driver variability, and full application speedup. Validation combines unit tests, reference outputs, randomized sizes, numerical tolerances, race and memory checking, API validation layers, shader or kernel sanitizers, static analysis, differential backends, trace capture, performance regression tests, long-duration stress, device-loss and out-of-memory injection, driver matrices, and responsive end-to-end tests. Explicit APIs require special attention to resource state, visibility, ownership transfers, fences, semaphores, barriers, and object lifetimes. Passing a visual demo does not prove synchronization or memory correctness. Portability has several layers: source language, intermediate representation, runtime API, device capability, numerical behavior, performance, and operational support. Code can compile everywhere yet perform poorly because subgroup width, cache, memory, compiler, or synchronization differs. Capability discovery, conformance tests, backend-specific tuning behind stable interfaces, reproducible toolchains, and graceful fallback make portability real. Vendor-specific paths can be valuable when their measured benefit exceeds maintenance and lock-in cost. GPU and accelerator software processes untrusted shaders, models, assets, and commands across shared drivers and memory. Validate sizes and formats, bound resource use, isolate DMA with platform protection, clear tenant state, sign and provenance build artifacts, control debug and profiling access, update drivers and firmware, and handle device loss without leaking data. Shader compilation and runtime code generation belong in the software supply chain and require dependency, cache, and artifact controls.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" rx="18" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Inside a GPU — the Parallel Engine</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">thousands of threads execute in lockstep warps across a grid of streaming multiprocessors</text>

  <!-- Die outline -->
  <rect x="30" y="62" width="430" height="340" rx="8" fill="#080d14" stroke="#2a3a4a" stroke-width="2"/>
  <text x="245" y="80" fill="#3a5a7a" font-size="10" text-anchor="middle">H100 SXM die (~814 mm²)</text>

  <!-- GPC cluster (4 GPCs shown) -->
  <!-- GPC 1 -->
  <rect x="44" y="90" width="200" height="145" rx="4" fill="#0b1524" stroke="#1e3a5f" stroke-width="1.2"/>
  <text x="144" y="106" fill="#60a5fa" font-size="10" text-anchor="middle" font-weight="600">GPC 0</text>

  <!-- SMs inside GPC (grid of small blocks) -->
  <g>
    <!-- Row 1: 4 SMs -->
    <rect x="52" y="112" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <text x="72" y="128" fill="#93c5fd" font-size="8" text-anchor="middle">SM</text>
    <rect x="96" y="112" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <text x="116" y="128" fill="#93c5fd" font-size="8" text-anchor="middle">SM</text>
    <rect x="140" y="112" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <text x="160" y="128" fill="#93c5fd" font-size="8" text-anchor="middle">SM</text>
    <rect x="184" y="112" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <text x="204" y="128" fill="#93c5fd" font-size="8" text-anchor="middle">SM</text>
    <!-- Row 2 -->
    <rect x="52" y="142" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <text x="72" y="158" fill="#93c5fd" font-size="8" text-anchor="middle">SM</text>
    <rect x="96" y="142" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <text x="116" y="158" fill="#93c5fd" font-size="8" text-anchor="middle">SM</text>
    <rect x="140" y="142" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <text x="160" y="158" fill="#93c5fd" font-size="8" text-anchor="middle">SM</text>
    <rect x="184" y="142" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <text x="204" y="158" fill="#93c5fd" font-size="8" text-anchor="middle">SM</text>
    <!-- Row 3 -->
    <rect x="52" y="172" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="96" y="172" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="140" y="172" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="184" y="172" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
  </g>
  <!-- Raster / Poly engine -->
  <rect x="52" y="202" width="172" height="18" rx="2" fill="#1a1520" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="138" y="214" fill="#c4b5fd" font-size="8" text-anchor="middle">Raster + PolyMorph Engine</text>

  <!-- GPC 2 -->
  <rect x="248" y="90" width="200" height="145" rx="4" fill="#0b1524" stroke="#1e3a5f" stroke-width="1.2"/>
  <text x="348" y="106" fill="#60a5fa" font-size="10" text-anchor="middle" font-weight="600">GPC 1</text>
  <!-- SM grid (simplified) -->
  <g>
    <rect x="256" y="112" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="300" y="112" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="344" y="112" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="388" y="112" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="256" y="142" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="300" y="142" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="344" y="142" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="388" y="142" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="256" y="172" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="300" y="172" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="344" y="172" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
    <rect x="388" y="172" width="40" height="26" rx="2" fill="#122040" stroke="#3b82f6" stroke-width="0.8"/>
  </g>
  <rect x="256" y="202" width="172" height="18" rx="2" fill="#1a1520" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="342" y="214" fill="#c4b5fd" font-size="8" text-anchor="middle">Raster + PolyMorph Engine</text>

  <!-- L2 Cache bar -->
  <rect x="44" y="240" width="404" height="22" rx="3" fill="#1a2a1a" stroke="#34d399" stroke-width="1.2"/>
  <text x="246" y="255" fill="#6ee7b7" font-size="10" text-anchor="middle">L2 Cache — 50 MB (shared across all SMs)</text>

  <!-- HBM stacks (bottom of die) -->
  <g>
    <rect x="44" y="272" width="60" height="55" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.2"/>
    <text x="74" y="292" fill="#fbbf24" font-size="9" text-anchor="middle">HBM3</text>
    <text x="74" y="305" fill="#a1701a" font-size="8" text-anchor="middle">stack 0</text>
    <text x="74" y="318" fill="#a1701a" font-size="8" text-anchor="middle">16 GB</text>

    <rect x="112" y="272" width="60" height="55" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.2"/>
    <text x="142" y="292" fill="#fbbf24" font-size="9" text-anchor="middle">HBM3</text>
    <text x="142" y="305" fill="#a1701a" font-size="8" text-anchor="middle">stack 1</text>

    <rect x="180" y="272" width="60" height="55" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.2"/>
    <text x="210" y="292" fill="#fbbf24" font-size="9" text-anchor="middle">HBM3</text>
    <text x="210" y="305" fill="#a1701a" font-size="8" text-anchor="middle">stack 2</text>

    <rect x="248" y="272" width="60" height="55" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.2"/>
    <text x="278" y="292" fill="#fbbf24" font-size="9" text-anchor="middle">HBM3</text>
    <text x="278" y="305" fill="#a1701a" font-size="8" text-anchor="middle">stack 3</text>

    <rect x="316" y="272" width="60" height="55" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.2"/>
    <text x="346" y="292" fill="#fbbf24" font-size="9" text-anchor="middle">HBM3</text>
    <text x="346" y="305" fill="#a1701a" font-size="8" text-anchor="middle">stack 4</text>

    <rect x="384" y="272" width="60" height="55" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.2"/>
    <text x="414" y="292" fill="#fbbf24" font-size="9" text-anchor="middle">HBM3</text>
    <text x="414" y="305" fill="#a1701a" font-size="8" text-anchor="middle">stack 5</text>
  </g>
  <text x="246" y="342" fill="#f59e0b" font-size="10" text-anchor="middle">80 GB total — 3.35 TB/s bandwidth</text>

  <!-- NVLink / PCIe interconnect -->
  <rect x="44" y="355" width="404" height="20" rx="3" fill="#1a1a2a" stroke="#a78bfa" stroke-width="1"/>
  <text x="246" y="369" fill="#c4b5fd" font-size="9.5" text-anchor="middle">NVLink 4.0 — 900 GB/s bidirectional (18 links)</text>

  <!-- RIGHT SIDE: SM internals (zoomed view) -->
  <rect x="478" y="62" width="260" height="260" rx="6" fill="#080d14" stroke="#3b82f6" stroke-width="1.5"/>
  <text x="608" y="82" fill="#93c5fd" font-size="12" text-anchor="middle" font-weight="600">Inside one SM</text>

  <!-- Warp schedulers -->
  <rect x="492" y="92" width="110" height="20" rx="2" fill="#1a2040" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="547" y="106" fill="#93c5fd" font-size="9" text-anchor="middle">Warp Scheduler x4</text>
  <rect x="610" y="92" width="110" height="20" rx="2" fill="#1a2040" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="665" y="106" fill="#93c5fd" font-size="9" text-anchor="middle">Dispatch Units</text>

  <!-- CUDA cores (grid of tiny squares) -->
  <text x="500" y="130" fill="#8b98a5" font-size="9">FP32 CUDA Cores (128):</text>
  <g fill="#1e3a5f" stroke="#3b82f6" stroke-width="0.4">
    <!-- 4 rows of 16 dots to represent 64 (half) -->
    <rect x="492" y="135" width="6" height="6"/><rect x="500" y="135" width="6" height="6"/><rect x="508" y="135" width="6" height="6"/><rect x="516" y="135" width="6" height="6"/><rect x="524" y="135" width="6" height="6"/><rect x="532" y="135" width="6" height="6"/><rect x="540" y="135" width="6" height="6"/><rect x="548" y="135" width="6" height="6"/><rect x="556" y="135" width="6" height="6"/><rect x="564" y="135" width="6" height="6"/><rect x="572" y="135" width="6" height="6"/><rect x="580" y="135" width="6" height="6"/><rect x="588" y="135" width="6" height="6"/><rect x="596" y="135" width="6" height="6"/><rect x="604" y="135" width="6" height="6"/><rect x="612" y="135" width="6" height="6"/>
    <rect x="492" y="143" width="6" height="6"/><rect x="500" y="143" width="6" height="6"/><rect x="508" y="143" width="6" height="6"/><rect x="516" y="143" width="6" height="6"/><rect x="524" y="143" width="6" height="6"/><rect x="532" y="143" width="6" height="6"/><rect x="540" y="143" width="6" height="6"/><rect x="548" y="143" width="6" height="6"/><rect x="556" y="143" width="6" height="6"/><rect x="564" y="143" width="6" height="6"/><rect x="572" y="143" width="6" height="6"/><rect x="580" y="143" width="6" height="6"/><rect x="588" y="143" width="6" height="6"/><rect x="596" y="143" width="6" height="6"/><rect x="604" y="143" width="6" height="6"/><rect x="612" y="143" width="6" height="6"/>
    <rect x="492" y="151" width="6" height="6"/><rect x="500" y="151" width="6" height="6"/><rect x="508" y="151" width="6" height="6"/><rect x="516" y="151" width="6" height="6"/><rect x="524" y="151" width="6" height="6"/><rect x="532" y="151" width="6" height="6"/><rect x="540" y="151" width="6" height="6"/><rect x="548" y="151" width="6" height="6"/><rect x="556" y="151" width="6" height="6"/><rect x="564" y="151" width="6" height="6"/><rect x="572" y="151" width="6" height="6"/><rect x="580" y="151" width="6" height="6"/><rect x="588" y="151" width="6" height="6"/><rect x="596" y="151" width="6" height="6"/><rect x="604" y="151" width="6" height="6"/><rect x="612" y="151" width="6" height="6"/>
    <rect x="492" y="159" width="6" height="6"/><rect x="500" y="159" width="6" height="6"/><rect x="508" y="159" width="6" height="6"/><rect x="516" y="159" width="6" height="6"/><rect x="524" y="159" width="6" height="6"/><rect x="532" y="159" width="6" height="6"/><rect x="540" y="159" width="6" height="6"/><rect x="548" y="159" width="6" height="6"/><rect x="556" y="159" width="6" height="6"/><rect x="564" y="159" width="6" height="6"/><rect x="572" y="159" width="6" height="6"/><rect x="580" y="159" width="6" height="6"/><rect x="588" y="159" width="6" height="6"/><rect x="596" y="159" width="6" height="6"/><rect x="604" y="159" width="6" height="6"/><rect x="612" y="159" width="6" height="6"/>
  </g>

  <!-- Tensor Cores -->
  <text x="500" y="182" fill="#8b98a5" font-size="9">Tensor Cores (4th gen, 4):</text>
  <g>
    <rect x="492" y="187" width="28" height="16" rx="2" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
    <text x="506" y="198" fill="#fbbf24" font-size="7" text-anchor="middle">TC</text>
    <rect x="524" y="187" width="28" height="16" rx="2" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
    <text x="538" y="198" fill="#fbbf24" font-size="7" text-anchor="middle">TC</text>
    <rect x="556" y="187" width="28" height="16" rx="2" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
    <text x="570" y="198" fill="#fbbf24" font-size="7" text-anchor="middle">TC</text>
    <rect x="588" y="187" width="28" height="16" rx="2" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
    <text x="602" y="198" fill="#fbbf24" font-size="7" text-anchor="middle">TC</text>
  </g>
  <text x="660" y="198" fill="#a1701a" font-size="8">FP8/FP16/TF32</text>

  <!-- Register file -->
  <rect x="492" y="210" width="230" height="16" rx="2" fill="#0f2a1f" stroke="#34d399" stroke-width="0.8"/>
  <text x="607" y="221" fill="#6ee7b7" font-size="9" text-anchor="middle">Register File — 256 KB (65536 x 32b)</text>

  <!-- Shared memory / L1 -->
  <rect x="492" y="232" width="230" height="16" rx="2" fill="#1a2a1a" stroke="#34d399" stroke-width="0.8"/>
  <text x="607" y="243" fill="#6ee7b7" font-size="9" text-anchor="middle">Shared Memory / L1 — 228 KB (configurable)</text>

  <!-- Load/Store units -->
  <rect x="492" y="254" width="110" height="14" rx="2" fill="#14202c" stroke="#3a4453" stroke-width="0.8"/>
  <text x="547" y="264" fill="#8b98a5" font-size="8" text-anchor="middle">Load/Store Units (32)</text>
  <rect x="610" y="254" width="110" height="14" rx="2" fill="#14202c" stroke="#3a4453" stroke-width="0.8"/>
  <text x="665" y="264" fill="#8b98a5" font-size="8" text-anchor="middle">SFU (16)</text>

  <!-- Execution model annotation -->
  <text x="608" y="290" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">1 warp = 32 threads (SIMT)</text>
  <text x="608" y="305" fill="#8b98a5" font-size="9.5" text-anchor="middle">4 warps active per clock → 128 threads/SM/cycle</text>
  <text x="608" y="318" fill="#6b7684" font-size="9" text-anchor="middle">latency hidden by warp switching, not caching</text>

  <!-- Key numbers (bottom right) -->
  <rect x="478" y="335" width="260" height="105" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="608" y="354" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">H100 SXM numbers</text>
  <text x="500" y="372" fill="#fbbf24" font-size="10">989 TFLOPS FP16 Tensor</text>
  <text x="500" y="388" fill="#fbbf24" font-size="10">1,979 TFLOPS FP8 Tensor</text>
  <text x="500" y="404" fill="#34d399" font-size="10">80 GB HBM3 @ 3.35 TB/s</text>
  <text x="500" y="420" fill="#60a5fa" font-size="10">132 SMs, 16,896 CUDA cores</text>
  <text x="500" y="436" fill="#8b98a5" font-size="10">700W TDP, TSMC 4N, 80B transistors</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">A GPU is a grid of SMs. Each SM is a grid of CUDA/Tensor cores. Parallelism hides latency instead of caches.</text>
</svg>

Selection, applications, and lifecycle ownership. GPUs fit wide parallel workloads and mature accelerator software. CPUs fit serial control and low-latency irregular work; NPUs fit supported inference; FPGAs fit custom deterministic pipelines; ASICs fit stable high-volume functions. AI, scientific simulation, molecular dynamics, finance, databases, media, rendering, cryptography, and engineering analysis use GPU computing. Requirements, representative traces, source, shaders or kernels, compiler and driver versions, generated binaries, architecture models, profiling baselines, device matrices, correctness evidence, performance budgets, known issues, rollout policy, telemetry, and deprecation decisions remain linked. APIs and silicon evolve at different rates, so teams define compatibility and fallback before deployment. Field measurements feed the next compiler, kernel, model, and hardware iteration without silently changing numerical or user-visible behavior. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.

Throughput orientation is the GPU’s central architectural bargain. A CPU spends substantial area and energy reducing the latency of a few instruction streams through large caches, branch prediction, speculation, and out-of-order scheduling. A GPU devotes more of the die to replicated arithmetic lanes, registers, schedulers, and bandwidth, accepting long single-thread latency when many independent groups can remain ready. This is not a claim that every GPU instruction is cheap. It means the workload must expose enough parallel work and regularity to amortize dispatch and tolerate latency. Comparing nominal core counts across CPUs, GPUs, or vendors is meaningless without execution width and instruction capability.

SIMT gives scalar-looking threads a grouped execution cost. NVIDIA CUDA groups threads into 32-lane warps, while AMD hardware uses architecture-dependent wavefront widths and other APIs describe subgroups. Each logical thread owns registers and may select its own branch, but hardware issues work across active lanes. If lanes choose different paths, the machine executes the required paths with masks, reducing useful work per issue slot. Divergence is expensive when it persists and divides active lanes; a short uniform branch or a branch between warps may be harmless. Correct optimization measures active-lane behavior instead of banning all conditionals.

The grid hierarchy defines both independence and cooperation. Kernels launch a grid of thread blocks or work-groups. Threads inside a block can share a low-latency scratchpad and synchronize at block scope; ordinary blocks must generally be independently schedulable because execution order is unspecified. Device-wide coordination normally requires kernel boundaries, cooperative-launch guarantees, atomics, or carefully defined memory semantics. Newer architectures may add cluster-level cooperation, but portability requires capability checks. Mapping one output element per thread is a starting point, not a law; persistent kernels, tiles, and producer-consumer specialization can be better.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,sans-serif">
<rect x="0" y="0" width="760" height="470" rx="18" fill="#0d1117"/><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Programming hierarchy maps onto finite hardware</text><text x="380" y="58" fill="#8b98a5" font-size="12" text-anchor="middle">Grids expose work; schedulers place resident groups on compute units</text>
<rect x="30" y="92" width="210" height="300" rx="12" fill="#161b22" stroke="#58a6ff" stroke-width="2"/><text x="135" y="122" fill="#79c0ff" font-size="14" font-weight="700" text-anchor="middle">Grid</text><g fill="#0d1117" stroke="#58a6ff"><rect x="62" y="155" width="58" height="48"/><rect x="148" y="155" width="58" height="48"/><rect x="62" y="230" width="58" height="48"/><rect x="148" y="230" width="58" height="48"/><rect x="62" y="305" width="58" height="48"/><rect x="148" y="305" width="58" height="48"/></g><text x="135" y="375" fill="#8b98a5" font-size="10" text-anchor="middle">independent blocks</text>
<path d="M245 242 H295" stroke="#3fb950" stroke-width="4"/><polygon points="295,242 283,235 283,249" fill="#3fb950"/><rect x="300" y="92" width="190" height="300" rx="12" fill="#161b22" stroke="#3fb950" stroke-width="2"/><text x="395" y="122" fill="#7ee787" font-size="14" font-weight="700" text-anchor="middle">Resident block</text><g fill="#1f6feb" stroke="#79c0ff"><circle cx="345" cy="175" r="15"/><circle cx="395" cy="175" r="15"/><circle cx="445" cy="175" r="15"/><circle cx="345" cy="230" r="15"/><circle cx="395" cy="230" r="15"/><circle cx="445" cy="230" r="15"/><circle cx="345" cy="285" r="15"/><circle cx="395" cy="285" r="15"/><circle cx="445" cy="285" r="15"/></g><text x="395" y="345" fill="#8b98a5" font-size="10" text-anchor="middle">threads share scratchpad and barrier</text>
<path d="M495 242 H545" stroke="#d29922" stroke-width="4"/><polygon points="545,242 533,235 533,249" fill="#d29922"/><rect x="550" y="92" width="180" height="300" rx="12" fill="#161b22" stroke="#d29922" stroke-width="2"/><text x="640" y="122" fill="#e3b341" font-size="14" font-weight="700" text-anchor="middle">SM or CU</text><rect x="580" y="155" width="120" height="42" rx="7" fill="#0d1117" stroke="#a371f7"/><text x="640" y="181" fill="#d2a8ff" font-size="11" text-anchor="middle">warp schedulers</text><rect x="580" y="220" width="120" height="42" rx="7" fill="#0d1117" stroke="#58a6ff"/><text x="640" y="246" fill="#79c0ff" font-size="11" text-anchor="middle">vector and matrix ALUs</text><rect x="580" y="285" width="120" height="42" rx="7" fill="#0d1117" stroke="#3fb950"/><text x="640" y="311" fill="#7ee787" font-size="11" text-anchor="middle">registers and local store</text></svg>

Occupancy is a resource result rather than a universal objective. Resident warps or wavefronts are limited by registers per thread, shared memory per block, threads per block, architectural block limits, and launch configuration. More resident groups can hide memory or pipeline latency, but forcing occupancy upward may increase spills, reduce useful instruction-level parallelism, or require smaller tiles with worse reuse. The relevant question is whether enough eligible work exists when a dependency stalls. Report achieved occupancy with stall reasons and throughput, not as a standalone score.

Register pressure couples compiler decisions to scheduling capacity. Registers hold the fastest thread-private state, yet each compute unit owns a finite physical file partitioned among resident groups. Aggressive unrolling, large tiles, many live accumulators, and inlining can increase reuse and simultaneously reduce residency. When allocation exceeds a threshold, occupancy drops discretely; when state spills, traffic goes to a much slower memory path often backed by global memory. Inspect generated code and spill counters. A lower source-level variable count does not guarantee fewer live registers after compiler optimization.

Latency hiding requires eligible independent instructions. Hardware can switch among ready warps quickly, but switching cannot help when every resident warp waits on the same dependency, barrier, cache miss, or throttled pipeline. Instruction-level parallelism within a thread, memory-level parallelism across outstanding transactions, and thread-level parallelism across warps work together. Long scoreboards can indicate data dependence or memory latency; not-selected cycles can indicate ample ready work; barrier stalls can identify phase imbalance. Counter names differ by vendor, so interpretation should follow the architecture’s documented scheduling model.

Coalescing converts lane requests into efficient memory transactions. Consecutive lanes accessing suitably aligned consecutive words allow the memory system to serve a warp with few transactions. A structure-of-arrays layout often coalesces a field better than an array of large structures, while pitched storage and padding can preserve row alignment. Misalignment, striding, scattering, and inactive lanes increase transferred bytes per useful byte. Caches may mask a pattern on one input without making it sound. Measure requested versus delivered bandwidth and test realistic working sets larger than cache.

The memory hierarchy is a set of scopes and policies, not one speed ladder. Registers are thread-private; shared memory or LDS is block-scoped and software-managed; L1 behavior may share capacity with scratchpad; L2 spans compute units; device memory supplies large capacity and bandwidth; host and peer memory add interconnect and coherence rules. Constant, texture, read-only, and specialized caches optimize particular access patterns. “Closer is faster” is incomplete because banking, occupancy, capacity, reuse, transaction size, and synchronization determine effective performance. State address space and visibility whenever discussing data placement.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,sans-serif">
<rect x="0" y="0" width="760" height="470" rx="18" fill="#0d1117"/><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">GPU memory hierarchy follows cooperation scope</text><text x="380" y="58" fill="#8b98a5" font-size="12" text-anchor="middle">Capacity grows outward while movement and synchronization costs accumulate</text>
<g text-anchor="middle"><rect x="270" y="88" width="220" height="55" rx="9" fill="#161b22" stroke="#f85149" stroke-width="2"/><text x="380" y="112" fill="#ff7b72" font-size="13" font-weight="700">Registers</text><text x="380" y="131" fill="#8b98a5" font-size="10">private to one thread</text><rect x="225" y="158" width="310" height="58" rx="9" fill="#161b22" stroke="#a371f7" stroke-width="2"/><text x="380" y="183" fill="#d2a8ff" font-size="13" font-weight="700">Shared memory or LDS and L1</text><text x="380" y="202" fill="#8b98a5" font-size="10">cooperation within a resident group</text><rect x="170" y="231" width="420" height="58" rx="9" fill="#161b22" stroke="#58a6ff" stroke-width="2"/><text x="380" y="256" fill="#79c0ff" font-size="13" font-weight="700">Shared last-level cache</text><text x="380" y="275" fill="#8b98a5" font-size="10">communication and reuse across compute units</text><rect x="115" y="304" width="530" height="58" rx="9" fill="#161b22" stroke="#3fb950" stroke-width="2"/><text x="380" y="329" fill="#7ee787" font-size="13" font-weight="700">HBM, GDDR, or device DRAM</text><text x="380" y="348" fill="#8b98a5" font-size="10">large capacity and high aggregate bandwidth</text><rect x="60" y="377" width="640" height="58" rx="9" fill="#161b22" stroke="#d29922" stroke-width="2"/><text x="380" y="402" fill="#e3b341" font-size="13" font-weight="700">Host, peer, storage, and network memory</text><text x="380" y="421" fill="#8b98a5" font-size="10">fabric topology, ownership, coherency, and transfer overhead</text></g></svg>

Shared-memory tiling trades traffic for explicit cooperation. A block loads a tile from global memory, synchronizes, reuses the tile across calculations, and writes results. Matrix multiplication, stencils, reductions, histograms, and transposes benefit when reuse exceeds load and barrier cost. Tile shape influences coalescing, halo overhead, bank mapping, registers, and occupancy. Asynchronous copies can overlap tile movement with computation when pipelines are correctly staged. Every producer-consumer handoff still needs a valid synchronization relation; fast scratchpad does not make races benign.

Bank conflicts serialize otherwise parallel scratchpad accesses. Shared memory or LDS is divided into banks that can serve independent addresses concurrently. When lanes address different words in the same bank, service may split into multiple transactions; broadcasts and architecture-specific multicast can be exceptions. Padding a transpose tile or changing the leading dimension often removes conflicts. Bank width and mapping are hardware properties, so folklore from one generation can mislead another. Use profiler counters and access arithmetic rather than treating all local-memory traffic as equally cheap.

The roofline model connects arithmetic intensity to achievable throughput. If a kernel performs $F$ operations while transferring $B$ bytes from a chosen memory level, arithmetic intensity is $I=F/B$. A simple ceiling is $P\leq\min(P_{peak},I\,BW)$, separating bandwidth-limited and compute-limited regions. The model becomes useful only when operations, bytes, precision, cache level, and attained ceilings are measured consistently. Low occupancy, dependencies, divergence, instruction mix, and launch overhead can keep performance below either roof. Hierarchical rooflines distinguish HBM, cache, and local-store reuse.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,sans-serif">
<rect x="0" y="0" width="760" height="470" rx="18" fill="#0d1117"/><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Roofline diagnosis separates traffic from arithmetic limits</text><text x="380" y="58" fill="#8b98a5" font-size="12" text-anchor="middle">Operational intensity must use measured work and bytes at a declared level</text>
<path d="M100 380 V95 M100 380 H700" stroke="#8b98a5" stroke-width="2"/><text x="35" y="240" fill="#8b98a5" font-size="11" text-anchor="middle" transform="rotate(-90 35 240)">attained operations per second</text><text x="400" y="430" fill="#8b98a5" font-size="11" text-anchor="middle">operations per byte</text><path d="M120 360 L435 135 H680" fill="none" stroke="#3fb950" stroke-width="5"/><text x="245" y="245" fill="#7ee787" font-size="12" transform="rotate(-35 245 245)">memory-bandwidth ceiling</text><text x="560" y="123" fill="#7ee787" font-size="12">compute ceiling</text><path d="M120 315 L365 140" fill="none" stroke="#58a6ff" stroke-width="3" stroke-dasharray="7 5"/><text x="185" y="290" fill="#79c0ff" font-size="10">cache roof</text><circle cx="240" cy="270" r="8" fill="#d29922"/><text x="252" y="263" fill="#e3b341" font-size="11">traffic-bound kernel</text><circle cx="525" cy="175" r="8" fill="#a371f7"/><text x="537" y="168" fill="#d2a8ff" font-size="11">underutilized compute</text><path d="M435 135 V380" stroke="#30363d" stroke-width="2" stroke-dasharray="5 5"/><text x="435" y="400" fill="#8b98a5" font-size="10" text-anchor="middle">ridge point</text></svg>

Arithmetic intensity should be improved before chasing nominal bandwidth. Fusion can keep intermediate values on chip, tiling can reuse operands, recomputation can cost less than storage, and a better algorithm can reduce total bytes. Yet fusion can increase registers and reduce scheduling freedom, while oversized tiles can lower occupancy. Compression and reduced precision shrink traffic only when conversion and accuracy costs are controlled. Compare end-to-end bytes and time, including temporary allocations and framework operations, because a fast fused kernel may expose a different system bottleneck.

Matrix engines accelerate structured operations under format constraints. Tensor Cores, AMD matrix fused multiply-add pipelines, and Intel XMX units perform small matrix operations at high rates for supported types and shapes. Libraries transform larger GEMM, convolution, and attention problems into tiled instruction sequences with staged data movement. Advertised tensor throughput assumes a particular precision, sparsity mode, accumulation rule, and operand utilization. Padding, layout conversion, scale computation, and epilogues consume real time. Validate numerical error and achieved instruction mix before crediting the peak number.

Mixed precision is a numerical method rather than a switch. Lower-precision storage and multiplication reduce bandwidth and increase matrix throughput, while wider accumulation, scaling, compensated reductions, or iterative refinement protect accuracy. FP16, BF16, TF32, FP8 families, integer formats, and vendor-specific encodings have different range and precision. Underflow, overflow, cancellation, nondeterministic reduction order, and optimizer sensitivity must be tested on representative tails, not only average loss. Report accuracy criteria beside speed and preserve a reference path for regression.

Synchronization must match the scope of communication. A block barrier coordinates participating threads in one group and usually establishes defined visibility for the relevant local memory; it does not synchronize unrelated blocks. Atomics serialize updates to one location according to an operation and memory order, but they do not automatically publish every surrounding access at every scope. Fences order visibility without guaranteeing rendezvous. Streams, queues, events, semaphores, and kernel boundaries express broader dependencies. Correct code identifies producer, consumer, address space, scope, order, and lifetime for every shared value.

Reductions expose both parallel structure and floating-point limits. Tree reductions replace a serial accumulation with logarithmic stages using subgroup exchange, shared memory, or specialized collectives. Associativity permits rearrangement over exact arithmetic, but floating-point addition is not associative, so block size and scheduling can change low bits. Compensated summation, pairwise order, wider accumulation, deterministic modes, and reproducible libraries trade performance for stability. Test adversarial magnitudes and cancellation. An atomic final step may be fast for few blocks and a contention bottleneck at scale.

Asynchronous pipelines overlap movement only when dependencies permit it. Double buffering lets one tile compute while another loads; streams can overlap copies and kernels when hardware engines, memory pinning, and independent work are available. NVIDIA’s Tensor Memory Accelerator and analogous engines reduce instruction and register overhead for structured transfers, but descriptors, alignment, barriers, and stage lifetimes remain part of correctness. A timeline with apparent overlap can still contend for HBM or copy engines. Measure elapsed critical path rather than summing isolated kernel savings.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,sans-serif">
<rect x="0" y="0" width="760" height="470" rx="18" fill="#0d1117"/><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">A tiled pipeline overlaps transfer and compute</text><text x="380" y="58" fill="#8b98a5" font-size="12" text-anchor="middle">Every stage still needs ownership, arrival, and reuse guarantees</text>
<g font-size="11" text-anchor="middle"><text x="75" y="120" fill="#8b98a5">time</text><path d="M110 115 H710" stroke="#30363d" stroke-width="2"/><text x="75" y="175" fill="#79c0ff">load</text><text x="75" y="250" fill="#7ee787">compute</text><text x="75" y="325" fill="#e3b341">store</text><rect x="120" y="145" width="130" height="48" rx="7" fill="#1f6feb"/><text x="185" y="174" fill="#fff">tile 0</text><rect x="255" y="145" width="130" height="48" rx="7" fill="#1f6feb"/><text x="320" y="174" fill="#fff">tile 1</text><rect x="390" y="145" width="130" height="48" rx="7" fill="#1f6feb"/><text x="455" y="174" fill="#fff">tile 2</text><rect x="255" y="220" width="130" height="48" rx="7" fill="#238636"/><text x="320" y="249" fill="#fff">tile 0</text><rect x="390" y="220" width="130" height="48" rx="7" fill="#238636"/><text x="455" y="249" fill="#fff">tile 1</text><rect x="525" y="220" width="130" height="48" rx="7" fill="#238636"/><text x="590" y="249" fill="#fff">tile 2</text><rect x="390" y="295" width="130" height="48" rx="7" fill="#9e6a03"/><text x="455" y="324" fill="#fff">tile 0</text><rect x="525" y="295" width="130" height="48" rx="7" fill="#9e6a03"/><text x="590" y="324" fill="#fff">tile 1</text></g><path d="M250 195 V215 M385 195 V215 M520 195 V215 M385 270 V290 M520 270 V290" stroke="#c9d1d9" stroke-width="2"/><rect x="145" y="380" width="520" height="44" rx="8" fill="#161b22" stroke="#30363d"/><text x="405" y="406" fill="#c9d1d9" font-size="11" text-anchor="middle">Overlap shortens the critical path only when engines and bandwidth are available.</text></svg>

Host-device transfer can dominate a kernel that benchmarks brilliantly. Discrete accelerators communicate through PCIe, CXL-class mechanisms, or proprietary fabrics with latency and bandwidth far below on-package memory. Batching, pinned buffers, zero-copy access, unified memory, prefetch, and long-lived device residency reduce explicit transfer cost under different conditions. Unified addressing simplifies ownership but page migration and faults can produce severe tails. Include allocation, initialization, transfer, launch, synchronization, and result use in the measurement boundary that matches the product.

Multi-GPU scaling is a topology and communication problem. Data parallelism exchanges gradients, model parallelism moves activations, pipeline parallelism sends stage boundaries, and domain decomposition exchanges halos. Ring, tree, recursive-doubling, and hierarchical collectives exploit different message sizes and fabrics. NVLink, Infinity Fabric, PCIe switches, and network adapters create nonuniform paths; GPUDirect-style transfers can avoid host staging when the platform supports them. Strong scaling eventually loses to communication and imbalance. Report useful work per device, collective time, topology, overlap, and end-to-end efficiency.

Profiling begins with a trustworthy timeline and ends with a causal test. CPU API traces reveal launch gaps, synchronization, allocation, and graph breaks; GPU timelines show queue overlap and dependencies; kernel counters report issue, stalls, cache, transactions, occupancy, and pipelines. Sampling reduces perturbation but may miss rare events, while instrumented replay changes execution conditions. Start from user-visible latency or throughput, locate the dominant interval, form a bottleneck hypothesis, change one mechanism, and confirm the predicted counter and outcome. Counter abundance does not substitute for an experiment.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,sans-serif">
<rect x="0" y="0" width="760" height="470" rx="18" fill="#0d1117"/><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Profile from application time to limiting mechanism</text><text x="380" y="58" fill="#8b98a5" font-size="12" text-anchor="middle">A counter matters only when a controlled change confirms its prediction</text>
<g text-anchor="middle" font-size="11"><rect x="30" y="105" width="150" height="86" rx="10" fill="#161b22" stroke="#58a6ff" stroke-width="2"/><text x="105" y="133" fill="#79c0ff" font-weight="700">Application</text><text x="105" y="158" fill="#c9d1d9">latency throughput</text><text x="105" y="176" fill="#8b98a5">tail and power</text><rect x="210" y="105" width="150" height="86" rx="10" fill="#161b22" stroke="#a371f7" stroke-width="2"/><text x="285" y="133" fill="#d2a8ff" font-weight="700">Timeline</text><text x="285" y="158" fill="#c9d1d9">launch transfer wait</text><text x="285" y="176" fill="#8b98a5">critical path</text><rect x="400" y="105" width="150" height="86" rx="10" fill="#161b22" stroke="#3fb950" stroke-width="2"/><text x="475" y="133" fill="#7ee787" font-weight="700">Kernel</text><text x="475" y="158" fill="#c9d1d9">traffic issue stalls</text><text x="475" y="176" fill="#8b98a5">resource use</text><rect x="580" y="105" width="150" height="86" rx="10" fill="#161b22" stroke="#d29922" stroke-width="2"/><text x="655" y="133" fill="#e3b341" font-weight="700">Experiment</text><text x="655" y="158" fill="#c9d1d9">change mechanism</text><text x="655" y="176" fill="#8b98a5">confirm outcome</text></g><g stroke="#30363d" stroke-width="3"><path d="M180 148 H210"/><path d="M360 148 H400"/><path d="M550 148 H580"/></g><path d="M655 200 V330 H105 V200" fill="none" stroke="#f85149" stroke-width="3" stroke-dasharray="7 6"/><text x="380" y="355" fill="#ff7b72" font-size="12" font-weight="700" text-anchor="middle">Retest the same representative workload</text><rect x="155" y="385" width="450" height="42" rx="8" fill="#161b22" stroke="#30363d"/><text x="380" y="411" fill="#c9d1d9" font-size="11" text-anchor="middle">If the predicted metric does not move, revise the hypothesis.</text></svg>

Benchmarking must separate peak, kernel, and application claims. Peak FLOPS multiply units, operations, and frequency under a supported instruction; peak bandwidth derives from memory rate and bus width. Microbenchmarks estimate sustainable ceilings, kernel benchmarks exercise one operator, and applications include orchestration and communication. Warmup, clock policy, thermal state, input distribution, compilation, autotuning, precision, sparsity, batch size, and synchronization all affect results. Compare equal accuracy and service constraints. A speedup needs the same baseline scope, not a selectively optimized numerator.

Power and thermal limits reshape sustained performance. Dynamic switching, leakage, HBM, interconnect, regulators, fans, and cooling infrastructure contribute to system power. Boost clocks consume thermal and electrical headroom, so a short benchmark can exceed steady-state throughput. Performance per watt depends on utilization: an oversized underused accelerator may waste idle and platform power, while batching can improve efficiency but violate latency. Log clocks, temperature, throttling, power cap, board power, host power, and cooling assumptions. Energy per completed valid result is often the more transferable metric.

Advanced packaging makes the accelerator a system of silicon. Large reticle-limited compute dies, chiplets, cache dies, HBM stacks, silicon interposers, bridges, substrates, and high-current power delivery determine bandwidth and yield. Microbumps and through-silicon vias shorten links but add thermal-mechanical and assembly constraints. HBM capacity and bandwidth scale through stacks and channels, while compute-to-memory balance determines usefulness. Yield, known-good-die test, warpage, hotspot coupling, repair, and package escape defects belong in architecture tradeoffs. The logical GPU cannot be evaluated independently of its package.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,sans-serif">
<rect x="0" y="0" width="760" height="470" rx="18" fill="#0d1117"/><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">GPU performance is assembled in the package</text><text x="380" y="58" fill="#8b98a5" font-size="12" text-anchor="middle">Compute, memory, fabric, power, and cooling set the sustainable system point</text>
<rect x="45" y="315" width="670" height="70" rx="10" fill="#161b22" stroke="#8b98a5" stroke-width="2"/><text x="380" y="358" fill="#c9d1d9" font-size="12" text-anchor="middle">organic substrate and package power delivery</text><rect x="95" y="270" width="570" height="38" rx="5" fill="#0b1524" stroke="#58a6ff" stroke-width="2"/><text x="380" y="295" fill="#79c0ff" font-size="11" text-anchor="middle">silicon interposer or advanced bridge fabric</text><rect x="255" y="125" width="250" height="138" rx="12" fill="#1f6feb" stroke="#79c0ff" stroke-width="3"/><text x="380" y="177" fill="#fff" font-size="16" font-weight="700" text-anchor="middle">GPU compute</text><text x="380" y="205" fill="#c9d1d9" font-size="11" text-anchor="middle">SM or CU arrays, caches, fabric</text><rect x="65" y="155" width="145" height="108" rx="9" fill="#9e6a03" stroke="#e3b341" stroke-width="3"/><rect x="550" y="155" width="145" height="108" rx="9" fill="#9e6a03" stroke="#e3b341" stroke-width="3"/><text x="137" y="200" fill="#fff" font-size="14" font-weight="700" text-anchor="middle">HBM stack</text><text x="622" y="200" fill="#fff" font-size="14" font-weight="700" text-anchor="middle">HBM stack</text><g stroke="#3fb950" stroke-width="4"><path d="M210 210 H255"/><path d="M505 210 H550"/></g><path d="M380 125 V85" stroke="#f85149" stroke-width="5"/><text x="380" y="78" fill="#ff7b72" font-size="11" text-anchor="middle">heat removal</text><g fill="#d2a8ff"><circle cx="290" cy="286" r="5"/><circle cx="330" cy="286" r="5"/><circle cx="370" cy="286" r="5"/><circle cx="410" cy="286" r="5"/><circle cx="450" cy="286" r="5"/></g><text x="380" y="425" fill="#8b98a5" font-size="10" text-anchor="middle">Package bandwidth, thermals, yield, and current delivery constrain product-level throughput.</text></svg>

Reliability includes silent errors as well as visible device loss. ECC protects selected memories and datapaths, page retirement and row remapping manage degrading storage, and telemetry reports corrected or uncorrected events. Cosmic rays, voltage margin, thermal stress, interconnect faults, and firmware defects can corrupt work or reset a device. Long distributed jobs amplify rare-event exposure. Use error injection where possible, validate checkpoints, detect stalled collectives, preserve diagnostic context, and define recovery. Redundant execution or algorithm-based fault tolerance may be justified when silent corruption cost exceeds overhead.

GPU isolation crosses software, memory, and DMA boundaries. Multi-tenant systems partition time, compute units, memory, or entire devices through processes, virtual functions, containers, and hardware instances. The threat model includes stale memory, side channels, malicious kernels, compiler inputs, firmware, peer access, and denial through unbounded work. IOMMUs, memory clearing, signed firmware, least-privilege device files, quotas, watchdogs, attestation, and controlled profiling reduce risk. Isolation claims must identify which caches, engines, fabrics, telemetry channels, and reset domains are actually partitioned.

A GPU is justified by workload evidence rather than accelerator fashion. Parallel fraction, batchability, locality, supported precision, software maturity, latency target, memory capacity, communication, utilization, power, capital cost, and engineering effort jointly determine value. Amdahl’s law limits whole-program speedup when serial work remains, while queueing makes throughput-oriented batching costly for tail latency. Compare CPU, GPU, NPU, FPGA, and ASIC paths with equal correctness and operational scope. Include development, portability, deployment, monitoring, and refresh costs alongside device price.

start: Define workload accuracy latency throughput power and cost envelope
baseline: Measure end to end on representative inputs and steady thermal state
timeline: Locate host transfer queue collective and kernel critical path
classify: Classify launch memory compute synchronization communication or thermal limit
model: Build roofline resource occupancy and topology hypothesis
change: Apply one algorithm layout tiling fusion precision or scheduling change
correct: Verify reference accuracy races bounds and synchronization
measure: Rerun identical workload and inspect predicted counters
works: Did the predicted mechanism and product metric improve?
retain: Keep change with capability guard regression budget and telemetry
revise: Reject explanation and investigate the next dominant interval
scale: Test irregular sizes sustained load and multi device behavior
deploy: Record validity envelope versions power state and fallback
start->baseline->timeline->classify->model->change->correct->measure->works
works->retain->scale->deploy
works->revise
revise->timeline

A credible GPU result connects useful work to the mechanism that made it faster. Preserve the workload, accuracy threshold, software stack, device configuration, thermal state, timeline, counter evidence, and controlled comparison. Peak specifications describe ceilings; occupancy and bandwidth are means; user-visible throughput, latency, energy, and correctness are outcomes. Read GPU architecture through a workload-and-data-movement lens rather than a core-count-and-peak-FLOPS lens.