Neural Network Accelerators are the specialized hardware processors designed to perform the matrix multiply-accumulate (MAC) operations that dominate neural network inference and training — achieving 10–100× better performance-per-watt than general-purpose CPUs and GPUs for AI workloads by exploiting the regular, predictable data flow of neural network computation through architectures like systolic arrays, dataflow processors, and near-memory compute engines.

Why Dedicated AI Hardware

• Neural networks are dominated by: Matrix multiply (GEMM), convolutions, element-wise ops, softmax.
• GEMM ≈ 80–95% of compute in transformers and CNNs.
• CPU: General-purpose, cache-heavy, branch-prediction logic wasteful for regular MAC streams.
• GPU: Good for parallel workloads but DRAM bandwidth bottleneck for inference (memory-bound).
• Accelerator: Eliminate general-purpose overhead → maximize MAC/watt → optimize data reuse.

Google TPU (Tensor Processing Unit)

• TPUv1 (2016): 256×256 systolic array, 8-bit multiply/32-bit accumulate.
• 92 tera-operations/second (TOPS), 28W — inference only.
• TPUv4 (2023): 460 TFLOPS (bfloat16), 4096 TPUv4 chips linked via mesh optical interconnect.
• TPUv5e: 197 TFLOPS per chip, optimized for inference cost efficiency.
• Architecture: Matrix Multiply Unit (MXU) = systolic array + HBM memory → weights loaded once, kept in MXU registers.

Systolic Array Architecture

Data flows through a grid of processing elements (PEs):

  Weight → PE(0,0) → PE(0,1) → PE(0,2)
              ↓          ↓          ↓
  Input  → PE(1,0) → PE(1,1) → PE(1,2)
              ↓          ↓          ↓
           PE(2,0) → PE(2,1) → PE(2,2) → Output (accumulate)

- Each PE: multiply input × weight + accumulate.
- Data flows: activations left→right, weights top→bottom.
- Each weight used N times (once per activation row) → enormous reuse.
- Result: Very high arithmetic intensity → stays compute-bound, not memory-bound.

Apple Neural Engine (ANE)

• Integrated into Apple Silicon (A-series, M-series chips).
• M4 ANE: 38 TOPS, optimized for int8 and float16 inference.
• Specializes in: Mobile Vision, NLP, on-device LLM inference (7B models on M3 Pro).
• Tight integration with CPU/GPU via unified memory → zero-copy tensor sharing.

Cerebras Wafer-Scale Engine (WSE)

• Single silicon wafer (46,225 mm²) containing 900,000 AI cores + 40GB SRAM.
• Eliminates off-chip memory bottleneck: All weights fit in on-chip SRAM for small models.
• 900K cores × 1 FLOP each = massive parallelism for sparse workloads.

Dataflow vs Systolic Architectures

Efficiency Metrics

• TOPS/W: Tera-operations per second per watt (efficiency).
• TOPS: Peak throughput (INT8 or FP16).
• TOPS/mm²: Silicon efficiency (cost proxy).
• Memory bandwidth: GB/s determines inference throughput for memory-bound workloads.

Neural network accelerators are the semiconductor manifestation of the AI revolution — just as the GPU transformed deep learning research by making matrix operations 100× faster than CPU, specialized AI chips like TPUs and NPUs are now making inference 10–100× more efficient than GPUs for specific workloads, enabling the deployment of trillion-parameter AI models in data centers and billion-parameter models on smartphones, while driving a new era of semiconductor design where AI workload requirements directly shape processor microarchitecture.