gpu (graphics processing unit)
**GPU (Graphics Processing Unit)** is the **massively parallel processor that has become the primary hardware accelerator for deep learning** — containing thousands of cores optimized for the matrix multiplications and tensor operations that dominate neural network training and inference, delivering 10-100x speedups over CPUs and fundamentally enabling the modern AI revolution from transformer models to generative AI through sheer computational throughput and high-bandwidth memory architectures.
**What Is a GPU?**
- **Definition**: A processor originally designed for rendering graphics that contains thousands of parallel compute cores capable of executing the same operation across massive data arrays simultaneously.
- **Why AI**: Neural networks are fundamentally matrix multiplication workloads — GPUs' SIMD (Single Instruction, Multiple Data) architecture maps perfectly to this computational pattern.
- **Market Dominance**: NVIDIA controls approximately 80-90% of the AI GPU market, with their CUDA ecosystem creating a powerful software moat.
- **Economic Impact**: GPU availability and cost are the primary bottleneck for AI research and deployment — GPU compute is the "new oil" of the AI era.
**Modern AI GPU Architecture**
| Component | Purpose | Example (H100) |
|-----------|---------|-----------------|
| **CUDA Cores** | General-purpose parallel computation | 16,896 cores |
| **Tensor Cores** | Specialized matrix multiply-accumulate units | 528 (4th gen) |
| **HBM (High Bandwidth Memory)** | High-speed memory for model weights and activations | 80GB HBM3 at 3.35 TB/s |
| **NVLink** | High-bandwidth GPU-to-GPU interconnect | 900 GB/s bidirectional |
| **Transformer Engine** | Automatic mixed-precision for transformers | FP8 support |
**Key NVIDIA GPU Generations for AI**
- **V100 (Volta, 2017)**: First Tensor Cores — established GPU as the AI training standard.
- **A100 (Ampere, 2020)**: Multi-Instance GPU (MIG), TF32 precision, dominant training GPU for 3 years.
- **H100 (Hopper, 2023)**: Transformer Engine with FP8, 3x A100 training performance, the chip that trained GPT-4-class models.
- **B200 (Blackwell, 2024)**: Next-generation architecture with further scaling of memory bandwidth and compute density.
**Why GPUs Matter for AI**
- **Training Speedup**: Operations that take weeks on CPUs complete in hours on GPU clusters — making large model training feasible.
- **Parallelism**: Thousands of cores execute matrix operations simultaneously, matching the inherently parallel nature of neural networks.
- **Memory Bandwidth**: HBM provides the bandwidth needed to feed data to compute cores fast enough to keep them utilized.
- **Ecosystem**: CUDA, cuDNN, NCCL, and frameworks like PyTorch provide optimized software stacks for GPU-accelerated deep learning.
- **Scaling**: Multi-GPU training with NVLink and InfiniBand enables training models across thousands of GPUs in large clusters.
**GPU Programming Ecosystem**
- **CUDA**: NVIDIA's parallel computing platform and programming model — the foundation of GPU-accelerated deep learning.
- **cuDNN**: GPU-accelerated library of primitives for deep neural networks (convolutions, normalizations, activations).
- **NCCL**: NVIDIA's library for multi-GPU and multi-node collective communication operations.
- **PyTorch/TensorFlow**: Deep learning frameworks that abstract CUDA programming into Python-level APIs.
- **TensorRT**: NVIDIA's inference optimization engine for deploying trained models with maximum GPU efficiency.
**Cloud GPU Access**
- **AWS**: P4d/P5 instances (A100/H100), SageMaker managed training.
- **Google Cloud**: A3 instances (H100), TPU alternatives for training.
- **Azure**: ND-series (A100/H100), integrated with Azure ML.
- **Lambda Cloud, CoreWeave, Together**: GPU-focused cloud providers with competitive pricing.
GPUs are **the engine powering the entire modern AI revolution** — providing the massive parallel compute throughput that makes training billion-parameter models feasible and inference at scale affordable, with GPU supply and innovation directly determining the pace of AI progress worldwide.