Machine learning hardware is the set of processors, memory systems, interconnects, and platforms optimized to train and serve neural networks. GPUs dominate large-scale AI because they combine high tensor throughput, HBM bandwidth, fast scale-up links, and mature software. TPUs and other custom ASICs trade flexibility for efficiency; FPGAs offer reconfigurable pipelines; CPUs coordinate irregular work. The winning platform is the whole usable system, not the chip with the largest TOPS number.

Training and inference emphasize different constraints. Training performs forward and backward passes, stores activations, communicates gradients, and benefits from high precision and massive cluster scaling. Inference must meet latency, throughput, availability, and cost targets while managing model weights and KV caches. Batch size improves utilization but increases delay. Edge inference prioritizes TOPS per watt, memory footprint, deterministic response, and integration with sensors.

The fundamental operation is multiply-accumulate, but feeding it is harder than instantiating it. Tensor cores, systolic arrays, vector units, and spatial dataflows reuse weights and activations locally. HBM supplies bulk bandwidth; SRAM buffers tiles; registers deliver operands each cycle. If reuse is poor, arithmetic waits on memory. Roofline analysis relates attainable operations per second to arithmetic intensity and memory bandwidth.

<svg viewBox="0 0 960 380" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="mlTitle mlDesc">
  <title id="mlTitle">Machine learning hardware taxonomy</title><desc id="mlDesc">ML workloads map to GPUs, custom ASICs, FPGAs, CPUs, and edge NPUs according to flexibility, efficiency, scale, and latency.</desc>
  <rect width="960" height="380" rx="18" fill="#1a1a17"/><text x="480" y="29" text-anchor="middle" fill="#f4f4f1" font-size="16" font-weight="700">ML hardware spans flexible processors to fixed spatial engines</text>
  <rect x="365" y="60" width="230" height="48" rx="9" fill="#292923" stroke="#2dd4bf" stroke-width="2"/><text x="480" y="90" text-anchor="middle" fill="#d5fffa" font-size="13" font-weight="700">Training and inference workloads</text>
  <g stroke="#2dd4bf" stroke-width="2"><path d="M480 108V140H105V165 M480 140H295V165 M480 140H480V165 M480 140H665V165 M480 140H855V165"/></g>
  <g fill="#292923" stroke-width="2"><rect x="35" y="165" width="140" height="110" rx="9" stroke="#6fbf6f"/><rect x="225" y="165" width="140" height="110" rx="9" stroke="#9a8adf"/><rect x="410" y="165" width="140" height="110" rx="9" stroke="#e0913a"/><rect x="595" y="165" width="140" height="110" rx="9" stroke="#2dd4bf"/><rect x="785" y="165" width="140" height="110" rx="9" stroke="#6fbf6f"/></g>
  <g text-anchor="middle" font-size="12" font-weight="700" fill="#eee"><text x="105" y="193">GPU</text><text x="295" y="193">Cloud ASIC</text><text x="480" y="193">FPGA</text><text x="665" y="193">Edge NPU</text><text x="855" y="193">CPU</text></g>
  <g text-anchor="middle" font-size="11" fill="#bbb"><text x="105" y="220">programmable</text><text x="105" y="239">massively parallel</text><text x="295" y="220">TPU / Trainium</text><text x="295" y="239">high efficiency</text><text x="480" y="220">custom datapath</text><text x="480" y="239">reconfigurable</text><text x="665" y="220">low power</text><text x="665" y="239">sensor integrated</text><text x="855" y="220">control + sparse</text><text x="855" y="239">ubiquitous host</text></g>
  <text x="480" y="318" text-anchor="middle" fill="#c9c2ef" font-size="12">Memory capacity, bandwidth, interconnect, compiler, and availability decide usable performance.</text>
  <text x="480" y="344" text-anchor="middle" fill="#aaa" font-size="11">Heterogeneous systems assign each graph region to the engine whose dataflow fits it best.</text>
</svg>

Precision is an architectural lever. FP32 remains important for selected accumulation and numerically sensitive work, while BF16 and FP16 are standard training formats. FP8 increases throughput and reduces bytes when scaling preserves accuracy. INT8 and INT4 are common for inference, and structured sparsity can skip work. Peak gains materialize only when kernels, memory layouts, calibration, and model quality support the format.

Memory capacity can be more important than TOPS. Parameters, optimizer state, gradients, activations, and KV cache must reside somewhere. HBM capacity determines partitioning and communication. Quantization, checkpointing, offload, and sharding extend model size at performance cost. Unified or coherent memory simplifies programming but does not eliminate physical bandwidth and page-migration limits.

Scale-up and scale-out interconnect determine distributed efficiency. Within a server, NVLink, Infinity Fabric, or a proprietary link supports high-bandwidth collectives and memory access. Across servers, InfiniBand or Ethernet carries all-reduce, all-to-all, and pipeline traffic. Expert parallel mixture-of-experts models stress network latency and tail behavior. Topology-aware libraries and congestion control are as important as link rate.

GPUs win through programmability and ecosystem. NVIDIA couples hardware with CUDA, cuDNN, NCCL, TensorRT, profilers, and broad framework support. AMD invests in ROCm and open software. GPUs efficiently run changing operators, custom kernels, graphics, simulation, and AI. Their generality costs control and scheduling overhead, but rapid model evolution often makes flexibility more valuable than theoretical ASIC efficiency.

Custom ASICs optimize a chosen dataflow and deployment environment. Google TPU systolic arrays, AWS Trainium and Inferentia, and other cloud designs align memory, arithmetic, interconnect, compiler, and fleet needs. Eliminating unused features can improve performance per watt and cost. The tradeoff is large nonrecurring engineering cost, long development cycles, and risk that models change before silicon arrives.

FPGAs occupy the adaptable middle. Configurable logic, DSP blocks, SRAM, and high-speed I/O implement streaming pipelines with deterministic latency. They suit rapidly changing protocols, pre/post-processing, network-attached inference, and modest-volume specialization. Clock speed and density trail ASICs, while programming remains harder than launching GPU kernels. High-level synthesis helps when designers still understand dataflow and timing.

Edge NPUs optimize constrained products. Smartphone, automotive, camera, and microcontroller accelerators share power and memory with the rest of the SoC. They fuse convolution, matrix, activation, resize, and compression operations and avoid DRAM traffic. Unsupported operators may fall back to CPU and erase gains. Toolchains therefore expose graph partitioning, quantization, profiling, and representative device emulation.

Scaling laws translate directly into infrastructure pressure. Larger models and datasets demand more compute, memory, communication, and energy, though algorithmic improvements change the slope. Long-context inference expands KV-cache traffic; sparse expert models increase network traffic; multimodal models add preprocessing. Hardware roadmaps must anticipate workload structure rather than extrapolate dense matrix FLOPs alone.

Performance metrics need context. TOPS states operations under a format and sparsity assumption. TOPS per watt may exclude host, memory, or cooling. Tokens per second depends on model, sequence length, batch, latency target, software, and quantization. Training throughput should include convergence quality and communication. Standard benchmarks help, but transparent configuration and total-system power are essential.

Reliability and availability matter at cluster scale. Thousands of devices expose marginal HBM, links, power supplies, firmware, and cooling events. ECC, link retry, health monitoring, checkpointing, workload replay, and spare capacity keep jobs productive. Silent numerical corruption is especially dangerous because it may appear as model instability. Fleet telemetry feeds screening and preventive replacement.

Software portability remains imperfect. Framework graph compilers, kernel DSLs, ONNX, MLIR, and vendor libraries reduce migration cost, but performance-sensitive code still depends on memory hierarchy and collective behavior. Compiler quality determines fusion, tiling, scheduling, precision, and communication overlap. A platform with lower peak math can win by supporting the model on day one.

Procurement is a system and supply decision. Buyers evaluate accelerator availability, HBM, network switches, optics, rack power, cooling, cloud contracts, software labor, and roadmap continuity. NVIDIA is dominant, while AMD, Google, AWS, Intel, and startups create alternatives. Multi-sourcing improves leverage but fragments engineering effort. Total useful tokens or training results per invested USD is more meaningful than chip price.

Machine learning hardware succeeds when it keeps data local, arithmetic busy, communication overlapped, and developers productive. No single architecture wins every phase. CPUs orchestrate, GPUs train and serve broad models, ASICs optimize stable high-volume workloads, FPGAs adapt specialized streams, and edge NPUs deliver low-power response. Heterogeneous co-design is the durable landscape.