tpu ai chip architecture google

**Google TPU Architecture: Systolic Array Matrix Computation — specialized tensor processor with data-reuse systolic fabric for efficient large-scale neural network inference and training on data centers and edge devices** **TPU Core Architecture Components** - **Systolic Array**: 128×128 MAC array (systolic execution — data flows through PEs), matrix multiply unit (MMU) for FP32/BF16/INT8 operations - **Unified Buffer**: 24 MB on-chip SRAM shared between systolic array and activation pipeline, avoids DRAM bandwidth bottleneck - **Activation Pipeline**: separates matrix multiply from activation functions (ReLU, GELU, Sigmoid), pipelined execution - **High-Bandwidth Memory (HBM)**: 2 TB/s aggregate for v4, compared to ~800 GB/s for GPU HBM **TPU Interconnect and Scaling** - **TPU Interconnect Mesh**: inter-chip communication for multi-TPU configurations (all-to-all via fabric), mesh or ring topology - **TPU Pods**: up to 1,024 TPUs networked together for large models, collective communication (allreduce) - **v1 to v4 Evolution**: v1 (2016, 8-bit integer only), v2 (TPU Pod 8×8 systolic), v3 (HBM stacking), v4 (enhanced HBM, improved peak throughput) **Performance Characteristics** - **Batch Size Dependency**: throughput scaling with batch size (large batches saturate compute, small batches underutilize) - **vs GPU**: TPU advantages (higher throughput per watt for inference), GPU advantages (flexibility, mixed precision, dynamic control flow) - **Google Cloud TPU Ecosystem**: Colab integration, TPU VMs, pricing model per-TPU **Applications and Limitations** - **Optimal Workloads**: dense tensor operations (CNNs, Transformers), large-scale training/inference - **Limitations**: fixed dataflow architecture (not suitable for irregular computation), control flow overhead, software maturity vs CUDA **Design Takeaways**: systolic array specialization enables 10-100× efficiency vs general CPU, massive on-chip memory reduces DRAM pressure, multi-TPU scaling via interconnect mesh for exascale training.