cuda

**CUDA (Compute Unified Device Architecture)** is NVIDIA's **parallel computing platform and programming model** that enables developers to use NVIDIA GPUs for general-purpose computation, including deep learning training and inference. CUDA is the foundation of the modern AI hardware ecosystem. **Why CUDA Dominates AI** - **First-Mover Advantage**: CUDA launched in 2007 and has had over 15 years of development, libraries, and ecosystem building. - **Software Ecosystem**: Decades of optimized libraries — **cuDNN** (deep learning primitives), **cuBLAS** (linear algebra), **NCCL** (multi-GPU communication), **TensorRT** (inference optimization). - **Framework Support**: **PyTorch** and **TensorFlow** are built on CUDA. Virtually all ML research code assumes CUDA. - **Developer Community**: Millions of developers, extensive documentation, tutorials, and Stack Overflow answers. **CUDA Architecture Concepts** - **Kernel**: A function executed in parallel by many GPU threads. - **Thread**: The smallest unit of execution. Threads are organized in **blocks**, and blocks form a **grid**. - **Streaming Multiprocessor (SM)**: The GPU's compute unit — each SM runs multiple thread blocks concurrently. - **Shared Memory**: Fast, on-chip memory shared between threads in a block. Critical for performance optimization. - **Global Memory**: The GPU's main memory (HBM/GDDR). High capacity but higher latency than shared memory. **CUDA for Deep Learning** - **cuDNN**: NVIDIA's deep learning library providing optimized implementations of convolutions, attention, normalization, activation functions, and other neural network operations. - **TensorRT**: Inference optimization engine that takes trained models and produces optimized CUDA kernels for production deployment. - **FlashAttention**: Custom CUDA kernel that implements attention more efficiently by optimizing memory access patterns. - **NCCL**: Multi-GPU and multi-node communication library for distributed training (AllReduce, AllGather, etc.). **CUDA Versions and Compatibility** - CUDA versions must be compatible with the GPU's **compute capability** (hardware generation) and the **NVIDIA driver** version. - **CUDA 12.x**: Current version, supporting Hopper (H100) and Ada Lovelace (RTX 4090) GPUs. - Framework compatibility: PyTorch releases are built against specific CUDA versions. **The CUDA Moat** CUDA's dominance is both technical and economic — the vast ecosystem of libraries, tools, and developer knowledge creates a **massive switching cost** that competitors (AMD ROCm, Intel oneAPI) struggle to overcome. This "CUDA moat" is NVIDIA's most valuable asset beyond the hardware itself.