AI Accelerator
An AI accelerator is a processor designed to execute the dense linear algebra at the heart of neural networks, mainly large matrix multiplications, far faster and more efficiently than a general-purpose CPU. The category includes graphics processing units (GPUs) repurposed for deep learning, purpose-built chips such as Google's Tensor Processing Unit (TPU), Amazon's Trainium and Inferentia, Graphcore's IPU, Cerebras's wafer-scale engine and Groq's inference chips, and the smaller on-device neural-processing-unit blocks in phones and laptops. All of them exploit the same fact: a neural network is mostly a vast number of regular multiply-accumulate operations that can run in parallel, so silicon spent on branch prediction and large caches can be reallocated to arithmetic.
GPUs became the default accelerator through a mix of timing and software rather than original design. Graphics rendering needs the same massively parallel arithmetic, and NVIDIA's CUDA platform, released in 2007, made GPUs programmable for general computation. In 2012 Krizhevsky, Sutskever and Hinton trained AlexNet on two GTX 580 GPUs and won the ImageNet challenge by a wide margin, which pushed the field toward GPU training. NVIDIA's Volta generation (the V100, 2017) added Tensor Cores, units that multiply small matrix tiles in mixed precision, and later generations added lower-precision formats such as FP8. A large part of NVIDIA's lasting advantage is the surrounding software, including CUDA, cuDNN and the NCCL communication library, which framework authors optimized against for years.
Google's TPU showed that dropping generality could pay off, and its core idea is the systolic array. Google first announced the TPU at its I/O conference in May 2016, saying it had already run in its data centers for roughly a year, and Jouppi and colleagues described the first generation in the 2017 paper "In-Datacenter Performance Analysis of a Tensor Processing Unit." That chip, built for inference, centered on a 256 × 256 matrix unit of 65,536 8-bit multiply-accumulate cells. In a systolic array, a concept associated with H. T. Kung and Charles Leiserson in the late 1970s, data flows rhythmically between neighboring cells, each of which multiplies, adds and passes values along, so operands are reused many times after a single memory read instead of being fetched repeatedly. Later TPU generations added training support, floating-point formats such as bfloat16, and pod-scale interconnects that link thousands of chips.
The binding constraint on modern accelerators is usually memory bandwidth, not arithmetic. This is the memory wall: compute capability has grown faster than the speed at which data can be delivered to it. The roofline model (Williams, Waterman and Patterson, 2009) captures this by showing that performance is capped either by peak arithmetic or by bandwidth times the number of operations performed per byte moved, and token-by-token generation in large language models sits on the bandwidth side because every generated token requires reading the model's weights and the kv-cache. High bandwidth memory (HBM) is the industry response: layers of DRAM dies are stacked vertically and connected through thousands of tiny vertical channels to the processor package. NVIDIA's A100 with 80 gigabytes of HBM2e offers roughly two terabytes per second, and the H100 with HBM3 offers more than three, a large multiple of conventional graphics memory. HBM is costly and limited in supply and capacity, which is why memory, not just compute, shapes the price and availability of accelerators and why techniques like quantization and flash-attention matter.
Accelerators are also limited by how well they talk to each other. Frontier models are trained across thousands of chips, so interconnects such as NVLink, Google's inter-chip interconnect, InfiniBand and Ethernet fabrics determine whether added chips translate into added speed, and collective operations such as all-reduce become a bottleneck at scale. The facility around the chips has constraints too, from power delivery to cooling, which are covered under data-center and energy-consumption.
Common misconceptions follow from treating the chip as a single score. Peak FLOPS figures assume ideal conditions and low-precision formats, while real utilization of a large training run is commonly only a fraction of peak, so comparisons should use measured benchmarks like MLPerf on the actual workload. A specialized chip is not automatically better than a GPU: it depends on whether a compiler can map the model well, whether the software ecosystem supports the framework, and whether the workload is training or inference. And the newest accelerator does not help if a model is memory-bound and the new part mainly adds arithmetic.
Read the AI accelerator as a machine for keeping thousands of multipliers fed: the arithmetic is the easy part, and each generation of GPU, TPU and HBM is mostly a new answer to the question of how to move data to it fast enough.