Google TPU Architecture: Systolic Array Matrix Computation — specialized tensor processor with data-reuse systolic fabric for efficient large-scale neural network inference and training on data centers and edge devices

TPU Core Architecture Components

• Systolic Array: 128×128 MAC array (systolic execution — data flows through PEs), matrix multiply unit (MMU) for FP32/BF16/INT8 operations
• Unified Buffer: 24 MB on-chip SRAM shared between systolic array and activation pipeline, avoids DRAM bandwidth bottleneck
• Activation Pipeline: separates matrix multiply from activation functions (ReLU, GELU, Sigmoid), pipelined execution
• High-Bandwidth Memory (HBM): 2 TB/s aggregate for v4, compared to ~800 GB/s for GPU HBM

TPU Interconnect and Scaling

• TPU Interconnect Mesh: inter-chip communication for multi-TPU configurations (all-to-all via fabric), mesh or ring topology
• TPU Pods: up to 1,024 TPUs networked together for large models, collective communication (allreduce)
• v1 to v4 Evolution: v1 (2016, 8-bit integer only), v2 (TPU Pod 8×8 systolic), v3 (HBM stacking), v4 (enhanced HBM, improved peak throughput)

Performance Characteristics

• Batch Size Dependency: throughput scaling with batch size (large batches saturate compute, small batches underutilize)
• vs GPU: TPU advantages (higher throughput per watt for inference), GPU advantages (flexibility, mixed precision, dynamic control flow)
• Google Cloud TPU Ecosystem: Colab integration, TPU VMs, pricing model per-TPU

Applications and Limitations

• Optimal Workloads: dense tensor operations (CNNs, Transformers), large-scale training/inference
• Limitations: fixed dataflow architecture (not suitable for irregular computation), control flow overhead, software maturity vs CUDA

Design Takeaways: systolic array specialization enables 10-100× efficiency vs general CPU, massive on-chip memory reduces DRAM pressure, multi-TPU scaling via interconnect mesh for exascale training.