machine learning accelerator npu
**Machine Learning Accelerator (NPU/TPU) Design** is the **computer architecture discipline that creates specialized hardware for neural network inference and training — implementing systolic arrays, matrix multiply engines, and dataflow architectures that deliver 10-1000× better performance-per-watt than general-purpose CPUs for the tensor operations (GEMM, convolution, activation) that dominate deep learning workloads**.
**Why ML Needs Specialized Hardware**
Neural networks are dominated by matrix multiplication: a single Transformer layer performs Q×K^T, attention×V, and two FFN GEMMs. A 70B parameter model executes ~140 TFLOPS per token. CPUs achieve <1 TFLOPS — too slow by >100×. GPUs improve to 50-300 TFLOPS but waste power on general-purpose hardware (branch prediction, cache hierarchy, out-of-order execution) unused by ML. ML accelerators strip unnecessary hardware and dedicate silicon to matrix math.
**Systolic Array Architecture**
The foundational ML accelerator structure (Google TPU, many NPUs):
- **2D Grid of PEs (Processing Elements)**: Each PE performs one multiply-accumulate (MAC) per cycle. Data flows through the array in a systolic (wave-like) pattern — inputs enter from edges, partial sums accumulate as data flows through PEs.
- **Weight-Stationary**: Weights are preloaded into PEs; input activations flow through. Each weight is used for many activations — maximum weight reuse.
- **Output-Stationary**: Partial sums accumulate in place; weights and activations flow through. Minimizes partial sum movement.
- **TPU v4**: 128×128 systolic array per core, BF16/INT8. 275 TFLOPS BF16 per chip. 4096 chips interconnected in a 3D torus (TPU pod) for distributed training.
**Dataflow Architecture**
Alternative to systolic arrays — compilers map the neural network's computation graph directly onto hardware:
- **Spatial Dataflow**: Each operation in the graph is mapped to a dedicated hardware block. Data flows between blocks without global memory access. Eliminates the von Neumann bottleneck. Examples: Graphcore IPU, Cerebras WSE.
- **Cerebras WSE-3**: Single wafer-scale chip (46,225 mm²) with 900,000 AI-optimized cores, 44 GB on-chip SRAM. Eliminates off-chip memory bandwidth bottleneck entirely — the entire model fits on-chip for models up to 24B parameters.
**Key Design Decisions**
- **Precision**: FP32 (training baseline), BF16/FP16 (standard training), FP8/INT8 (inference), INT4/INT2 (aggressive quantized inference). Lower precision = more MACs per mm² and per watt. Hardware must support mixed-precision accumulation (FP8 multiply, FP32 accumulate).
- **Memory Hierarchy**: On-chip SRAM bandwidth >> HBM bandwidth. Maximizing on-chip buffer size reduces HBM traffic. The ratio of compute FLOPS to memory bandwidth (arithmetic intensity) determines whether a workload is compute-bound or memory-bound.
- **Interconnect**: Multi-chip scaling requires high-bandwidth, low-latency interconnect. NVLink (900 GB/s GPU-GPU), TPU ICI (inter-chip interconnect), and custom D2D links enable distributed training across hundreds of chips.
**Energy Efficiency**
| Chip | Process | Peak TOPS (INT8) | TDP | TOPS/W |
|------|---------|-----------------|-----|--------|
| Google TPU v5e | 7nm (inferred) | 400 | 200W | 2.0 |
| NVIDIA H100 | TSMC 4N | 3,958 | 700W | 5.7 |
| Apple M4 Neural Engine | TSMC 3nm | 38 | 10W | 3.8 |
| Qualcomm Hexagon NPU | 4nm | 75 | 15W | 5.0 |
ML Accelerator Design is **the purpose-built silicon that makes practical AI inference and training computationally and economically feasible** — delivering orders of magnitude better efficiency than general-purpose processors by dedicating every transistor to the mathematical operations that neural networks actually need.