neural network accelerator
**Neural Network Accelerators** are the **specialized hardware processors designed to perform the matrix multiply-accumulate (MAC) operations that dominate neural network inference and training** — achieving 10–100× better performance-per-watt than general-purpose CPUs and GPUs for AI workloads by exploiting the regular, predictable data flow of neural network computation through architectures like systolic arrays, dataflow processors, and near-memory compute engines.
**Why Dedicated AI Hardware**
- Neural networks are dominated by: Matrix multiply (GEMM), convolutions, element-wise ops, softmax.
- GEMM ≈ 80–95% of compute in transformers and CNNs.
- CPU: General-purpose, cache-heavy, branch-prediction logic wasteful for regular MAC streams.
- GPU: Good for parallel workloads but DRAM bandwidth bottleneck for inference (memory-bound).
- Accelerator: Eliminate general-purpose overhead → maximize MAC/watt → optimize data reuse.
**Google TPU (Tensor Processing Unit)**
- TPUv1 (2016): 256×256 systolic array, 8-bit multiply/32-bit accumulate.
- 92 tera-operations/second (TOPS), 28W — inference only.
- TPUv4 (2023): 460 TFLOPS (bfloat16), 4096 TPUv4 chips linked via mesh optical interconnect.
- TPUv5e: 197 TFLOPS per chip, optimized for inference cost efficiency.
- Architecture: Matrix Multiply Unit (MXU) = systolic array + HBM memory → weights loaded once, kept in MXU registers.
**Systolic Array Architecture**
```
Data flows through a grid of processing elements (PEs):
Weight → PE(0,0) → PE(0,1) → PE(0,2)
↓ ↓ ↓
Input → PE(1,0) → PE(1,1) → PE(1,2)
↓ ↓ ↓
PE(2,0) → PE(2,1) → PE(2,2) → Output (accumulate)
- Each PE: multiply input × weight + accumulate.
- Data flows: activations left→right, weights top→bottom.
- Each weight used N times (once per activation row) → enormous reuse.
- Result: Very high arithmetic intensity → stays compute-bound, not memory-bound.
```
**Apple Neural Engine (ANE)**
- Integrated into Apple Silicon (A-series, M-series chips).
- M4 ANE: 38 TOPS, optimized for int8 and float16 inference.
- Specializes in: Mobile Vision, NLP, on-device LLM inference (7B models on M3 Pro).
- Tight integration with CPU/GPU via unified memory → zero-copy tensor sharing.
**Cerebras Wafer-Scale Engine (WSE)**
- Single silicon wafer (46,225 mm²) containing 900,000 AI cores + 40GB SRAM.
- Eliminates off-chip memory bottleneck: All weights fit in on-chip SRAM for small models.
- 900K cores × 1 FLOP each = massive parallelism for sparse workloads.
**Dataflow vs Systolic Architectures**
| Approach | Data Movement | Good For |
|----------|--------------|----------|
| Systolic array (TPU) | Regular grid flow | Dense matrix multiply |
| Dataflow (Graphcore) | Compute → compute | Graph-structured workloads |
| Near-memory (Samsung HBM-PIM) | Compute in memory | Memory-bound ops |
| Spatial (Sambanova) | Reconfigurable | Large batches, variable graphs |
**Efficiency Metrics**
- **TOPS/W**: Tera-operations per second per watt (efficiency).
- **TOPS**: Peak throughput (INT8 or FP16).
- **TOPS/mm²**: Silicon efficiency (cost proxy).
- **Memory bandwidth**: GB/s determines inference throughput for memory-bound workloads.
Neural network accelerators are **the semiconductor manifestation of the AI revolution** — just as the GPU transformed deep learning research by making matrix operations 100× faster than CPU, specialized AI chips like TPUs and NPUs are now making inference 10–100× more efficient than GPUs for specific workloads, enabling the deployment of trillion-parameter AI models in data centers and billion-parameter models on smartphones, while driving a new era of semiconductor design where AI workload requirements directly shape processor microarchitecture.