CUDA is NVIDIA's platform for general-purpose computing on GPUs: a programming model, a compiler and runtime, and a deep stack of libraries that together let software harness thousands of GPU cores. It is the single biggest reason NVIDIA — rather than any competitor with comparable silicon — owns AI compute. The hardware is replicable; the fifteen-plus years of CUDA software and developer habit built on top of it are not. The diagram shows that stack, and where the moat actually lives.\n\n``svg\n\n``\n\nThe programming model is SIMT. A CUDA program launches a kernel as a grid of thread blocks. Each block is scheduled onto one streaming multiprocessor, and its threads execute in warps of 32 that step through instructions in lockstep. The developer writes scalar code describing what one thread does, and the hardware replicates it across thousands of threads — a model that maps cleanly onto the GPU's wide, parallel execution units.\n\nCUDA cores and Tensor Cores do different jobs. CUDA cores are the general-purpose scalar and vector lanes that handle elementwise operations, activations, reductions, indexing, and control-heavy code. Tensor Cores are dedicated matrix-multiply-accumulate units. A well-tuned kernel keeps both busy; the classic failure mode is starving the Tensor Cores by spending too much time in elementwise glue, which is why kernel fusion — combining many small operations into one launch — is such a common optimization.\n\nThe library stack is the real product. cuBLAS for dense linear algebra, cuDNN for convolution and attention primitives, CUTLASS for templated GEMM, NCCL for multi-GPU collectives, plus cuFFT, Thrust, and more. Frameworks like PyTorch and JAX call into these under the hood, so most AI engineers get hand-tuned GPU performance without ever writing a line of CUDA themselves. That inherited performance is what makes the ecosystem sticky.\n\nPTX and forward compatibility deepen the lock-in. nvcc compiles CUDA to PTX, a virtual instruction set, which is then lowered to architecture-specific SASS. Because PTX is stable across generations, code written years ago still runs on new GPUs — an ABI guarantee that protects customers' software investment and, in turn, protects NVIDIA.\n\nThe challengers all attack the software, not the silicon. AMD's ROCm/HIP, Intel's oneAPI/SYCL, OpenAI's Triton (which lets you write GPU kernels in Python), and framework compilers like torch.compile and XLA are each attempts to make GPU code portable off CUDA. They are gaining ground, but the default path in AI still bottoms out in CUDA, and the switching cost is the accumulated ecosystem rather than any single instruction set.\n\n| Layer | Examples | Role |\n|---|---|---|\n| Frameworks | PyTorch, JAX, TensorFlow | model authoring |\n| Primitive libraries | cuDNN, cuBLAS, CUTLASS, NCCL | tuned kernels |\n| Runtime and language | CUDA C++, PTX, nvcc | kernel compilation |\n| Driver | NVIDIA kernel driver | hardware access |\n| Portability challengers | ROCm/HIP, SYCL, Triton, XLA | break the lock-in |\n\nRead CUDA through an ecosystem-lock-in lens rather than a language lens: what a competitor has to overcome is not the syntax of a kernel but fifteen years of tuned libraries, framework integrations, and developer muscle memory that all assume CUDA underneath. Every serious challenge — ROCm, SYCL, Triton, XLA — is ultimately an attempt to make that accumulated software portable, which is why the moat has held even as rival silicon caught up on raw FLOPS.\n