A GPU (graphics processing unit) is the workhorse of modern AI: a massively parallel processor that turns deep learning's core operation — multiplying huge matrices — into tens of thousands of arithmetic operations running at once. It began as a triangle-rasterizer for games, but the same wide, throughput-first design that shades millions of pixels turned out to be exactly what training and running neural networks needs. The diagram below is the anatomy: thousands of cores, and the bandwidth hierarchy that keeps them fed.\n\n``svg\n\n``\n\nA GPU is a throughput machine, not a latency machine. A CPU spends its transistors on a few powerful cores with deep caches and branch predictors, optimized to finish one thread as fast as possible. A GPU makes the opposite bet: thousands of simple cores grouped into streaming multiprocessors (SMs), running the same instruction across many data elements at once. This SIMT (single-instruction, multiple-thread) design is a poor fit for branchy, sequential code and a perfect fit for the dense linear algebra at the heart of neural networks.\n\nTensor Cores are why GPUs dominate AI. Since the Volta generation in 2017, NVIDIA data-center GPUs carry dedicated matrix-multiply units that perform a full small matrix multiply-accumulate every clock, at reduced precision such as FP16, BF16, and now FP8. The overwhelming majority of a transformer's FLOPs run on these units; the general-purpose CUDA cores handle the surrounding elementwise math, activations, and control. A workload that cannot keep the Tensor Cores busy leaves most of the chip's arithmetic power idle.\n\nThe memory hierarchy is the real constraint. Registers, shared memory and L1, L2, HBM, NVLink, and the network each drop roughly an order of magnitude in bandwidth as data moves farther from the cores. Peak arithmetic only materializes if operands stay high in that hierarchy, which is exactly what techniques like kernel fusion, tiling, and FlashAttention are for — they trade recomputation for staying on-chip and out of slow memory.\n\nScaling out turns one GPU into a cluster. NVLink and NVSwitch bind GPUs into a tightly coupled node; InfiniBand or high-speed Ethernet stitches nodes into a pod. Data, tensor, and pipeline parallelism then spread a model across the fabric. At frontier scale, communication bandwidth and memory capacity — not raw FLOPs — usually set the training time, which is why interconnect is now a first-class part of GPU system design.\n\nThe CUDA software stack is the moat. The reason NVIDIA rather than a competitor owns AI compute is not only the silicon but the fifteen-plus years of CUDA libraries, framework integrations, and developer habit layered on top of it. Rival accelerators can match FLOPs; matching the ecosystem is the hard part.\n\n| Data-center GPU | Year | Memory | Landmark |\n|---|---|---|---|\n| V100 | 2017 | 16–32 GB HBM2 | first Tensor Cores |\n| A100 | 2020 | 40–80 GB HBM2e | TF32, MIG, structured sparsity |\n| H100 | 2022 | 80 GB HBM3 | FP8, Transformer Engine |\n| Blackwell B200 | 2024 | up to 192 GB HBM3e | FP4, dual-die package |\n\nRead a GPU through a bandwidth-and-occupancy lens rather than a TFLOPS lens: the peak arithmetic rate on the datasheet only matters if you can keep the Tensor Cores fed, so the numbers that actually set training and serving throughput are memory bandwidth, interconnect bandwidth, and how much of the chip stays busy. Every optimization that matters — mixed precision, kernel fusion, FlashAttention, tensor and pipeline parallelism — is a different way to move less data and keep more cores working.\n