CUDA (Compute Unified Device… | ChipFoundryServices

Home› Knowledge Base› CUDA (Compute Unified Device Architecture)

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform and API for GPU programming — enabling developers to leverage GPU hardware for general-purpose computing, CUDA is the foundation of modern AI/ML frameworks with extensive ecosystem support through cuDNN, cuBLAS, and integration with PyTorch and TensorFlow.

What Is CUDA?

Definition: Programming model and API for NVIDIA GPU computing.
Purpose: General-purpose GPU (GPGPU) programming.
Language: C/C++ extensions with CUDA-specific syntax.
Ecosystem: Libraries, tools, frameworks built on CUDA.

Why CUDA Dominates AI

First Mover: Launched 2006, decade+ head start.
Ecosystem: Massive library and framework support.
Optimization: Highly tuned for NVIDIA hardware.
Community: Large developer base and resources.
Lock-in: Most AI code assumes CUDA.

CUDA Architecture Concepts

Execution Model:

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">CUDA — The GPU Programming Model</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">launch a kernel: millions of threads organized into blocks, scheduled onto SMs in warps of 32</text>

  <!-- === LEFT: Thread hierarchy (Grid → Block → Thread) === -->
  <rect x="30" y="62" width="340" height="250" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="200" y="80" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Thread Hierarchy (software view)</text>

  <!-- Grid outline -->
  <rect x="45" y="90" width="310" height="210" rx="5" fill="none" stroke="#76b900" stroke-width="1.2" stroke-dasharray="4,2"/>
  <text x="200" y="104" fill="#76b900" font-size="10" text-anchor="middle" font-weight="600">Grid (all blocks in one kernel launch)</text>

  <!-- Block (0,0) -->
  <rect x="55" y="112" width="140" height="80" rx="4" fill="#0b1524" stroke="#60a5fa" stroke-width="1.2"/>
  <text x="125" y="126" fill="#93c5fd" font-size="9" text-anchor="middle" font-weight="600">Block (0,0)</text>
  <!-- Threads inside block as tiny squares -->
  <g fill="#122040" stroke="#3b82f6" stroke-width="0.4">
    <rect x="63" y="132" width="8" height="8" rx="0.5"/><rect x="73" y="132" width="8" height="8" rx="0.5"/><rect x="83" y="132" width="8" height="8" rx="0.5"/><rect x="93" y="132" width="8" height="8" rx="0.5"/>
    <rect x="103" y="132" width="8" height="8" rx="0.5"/><rect x="113" y="132" width="8" height="8" rx="0.5"/><rect x="123" y="132" width="8" height="8" rx="0.5"/><rect x="133" y="132" width="8" height="8" rx="0.5"/>
    <rect x="63" y="142" width="8" height="8" rx="0.5"/><rect x="73" y="142" width="8" height="8" rx="0.5"/><rect x="83" y="142" width="8" height="8" rx="0.5"/><rect x="93" y="142" width="8" height="8" rx="0.5"/>
    <rect x="103" y="142" width="8" height="8" rx="0.5"/><rect x="113" y="142" width="8" height="8" rx="0.5"/><rect x="123" y="142" width="8" height="8" rx="0.5"/><rect x="133" y="142" width="8" height="8" rx="0.5"/>
    <rect x="63" y="152" width="8" height="8" rx="0.5"/><rect x="73" y="152" width="8" height="8" rx="0.5"/><rect x="83" y="152" width="8" height="8" rx="0.5"/><rect x="93" y="152" width="8" height="8" rx="0.5"/>
    <rect x="103" y="152" width="8" height="8" rx="0.5"/><rect x="113" y="152" width="8" height="8" rx="0.5"/><rect x="123" y="152" width="8" height="8" rx="0.5"/><rect x="133" y="152" width="8" height="8" rx="0.5"/>
    <rect x="63" y="162" width="8" height="8" rx="0.5"/><rect x="73" y="162" width="8" height="8" rx="0.5"/><rect x="83" y="162" width="8" height="8" rx="0.5"/><rect x="93" y="162" width="8" height="8" rx="0.5"/>
    <rect x="103" y="162" width="8" height="8" rx="0.5"/><rect x="113" y="162" width="8" height="8" rx="0.5"/><rect x="123" y="162" width="8" height="8" rx="0.5"/><rect x="133" y="162" width="8" height="8" rx="0.5"/>
  </g>
  <!-- Warp highlight (first row) -->
  <rect x="62" y="131" width="80" height="10" fill="#76b900" opacity="0.15" rx="1"/>
  <text x="125" y="180" fill="#6b7684" font-size="7.5" text-anchor="middle">32 threads (warp 0)</text>
  <text x="125" y="190" fill="#6b7684" font-size="7" text-anchor="middle">256 threads/block typical</text>

  <!-- Block (1,0) -->
  <rect x="205" y="112" width="140" height="80" rx="4" fill="#0b1524" stroke="#60a5fa" stroke-width="1.2"/>
  <text x="275" y="126" fill="#93c5fd" font-size="9" text-anchor="middle" font-weight="600">Block (1,0)</text>
  <g fill="#122040" stroke="#3b82f6" stroke-width="0.4">
    <rect x="213" y="132" width="8" height="8" rx="0.5"/><rect x="223" y="132" width="8" height="8" rx="0.5"/><rect x="233" y="132" width="8" height="8" rx="0.5"/><rect x="243" y="132" width="8" height="8" rx="0.5"/>
    <rect x="253" y="132" width="8" height="8" rx="0.5"/><rect x="263" y="132" width="8" height="8" rx="0.5"/><rect x="273" y="132" width="8" height="8" rx="0.5"/><rect x="283" y="132" width="8" height="8" rx="0.5"/>
    <rect x="213" y="142" width="8" height="8" rx="0.5"/><rect x="223" y="142" width="8" height="8" rx="0.5"/><rect x="233" y="142" width="8" height="8" rx="0.5"/><rect x="243" y="142" width="8" height="8" rx="0.5"/>
    <rect x="253" y="142" width="8" height="8" rx="0.5"/><rect x="263" y="142" width="8" height="8" rx="0.5"/><rect x="273" y="142" width="8" height="8" rx="0.5"/><rect x="283" y="142" width="8" height="8" rx="0.5"/>
  </g>

  <!-- More blocks (row 2) -->
  <rect x="55" y="200" width="140" height="40" rx="4" fill="#0b1524" stroke="#60a5fa" stroke-width="0.8" opacity="0.6"/>
  <text x="125" y="224" fill="#60a5fa" font-size="8" text-anchor="middle" opacity="0.7">Block (0,1)</text>
  <rect x="205" y="200" width="140" height="40" rx="4" fill="#0b1524" stroke="#60a5fa" stroke-width="0.8" opacity="0.6"/>
  <text x="275" y="224" fill="#60a5fa" font-size="8" text-anchor="middle" opacity="0.7">Block (1,1)</text>

  <!-- Dots for more -->
  <text x="200" y="260" fill="#6b7684" font-size="9" text-anchor="middle">... thousands of blocks</text>
  <text x="200" y="278" fill="#76b900" font-size="9" text-anchor="middle">kernel<<<grid, block>>>(args);</text>
  <text x="200" y="292" fill="#6b7684" font-size="8" text-anchor="middle">each thread gets: threadIdx, blockIdx, blockDim</text>

  <!-- === RIGHT: Memory hierarchy === -->
  <rect x="385" y="62" width="345" height="170" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="557" y="80" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">CUDA Memory Model</text>

  <!-- Registers (per thread) -->
  <rect x="400" y="92" width="140" height="22" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="470" y="107" fill="#fbbf24" font-size="9" text-anchor="middle">Registers (per thread)</text>
  <text x="560" y="107" fill="#6b7684" font-size="8">fastest, private</text>

  <!-- Shared memory (per block) -->
  <rect x="400" y="120" width="140" height="22" rx="3" fill="#14261f" stroke="#34d399" stroke-width="1"/>
  <text x="470" y="135" fill="#6ee7b7" font-size="9" text-anchor="middle">Shared Memory (block)</text>
  <text x="560" y="135" fill="#6b7684" font-size="8">~100 KB, user-managed</text>

  <!-- L1/L2 cache -->
  <rect x="400" y="148" width="140" height="22" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="470" y="163" fill="#c4b5fd" font-size="9" text-anchor="middle">L1 / L2 Cache</text>
  <text x="560" y="163" fill="#6b7684" font-size="8">HW-managed</text>

  <!-- Global memory (HBM) -->
  <rect x="400" y="176" width="140" height="22" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1"/>
  <text x="470" y="191" fill="#93c5fd" font-size="9" text-anchor="middle">Global Memory (HBM)</text>
  <text x="560" y="191" fill="#6b7684" font-size="8">80 GB, slowest, all threads</text>

  <!-- Speed indicator -->
  <line x1="392" y1="96" x2="392" y2="195" stroke="#f87171" stroke-width="0.8"/>
  <text x="388" y="145" fill="#f87171" font-size="7" text-anchor="end" transform="rotate(-90,388,145)">slower ↓</text>

  <text x="557" y="218" fill="#8b98a5" font-size="8.5" text-anchor="middle">Key: minimize global memory access (coalesce, tile into shared)</text>
  <text x="557" y="232" fill="#6b7684" font-size="8" text-anchor="middle">Occupancy: keep many warps active to hide memory latency</text>

  <!-- === BOTTOM: Software stack + ecosystem === -->
  <rect x="385" y="242" width="345" height="70" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="557" y="260" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">CUDA Ecosystem (the moat)</text>
  <text x="405" y="278" fill="#8b98a5" font-size="9">cuBLAS (linear algebra) · cuDNN (deep learning)</text>
  <text x="405" y="294" fill="#8b98a5" font-size="9">NCCL (multi-GPU comms) · TensorRT (inference)</text>
  <text x="405" y="310" fill="#6b7684" font-size="8.5">cuFFT, Thrust, CUB, cutlass — 15+ years of libraries</text>

  <!-- Bottom key concept -->
  <rect x="30" y="325" width="340" height="90" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="343" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">SIMT: Single Instruction, Multiple Threads</text>
  <text x="50" y="363" fill="#8b98a5" font-size="9">• 32 threads in a warp execute the same instruction</text>
  <text x="50" y="379" fill="#8b98a5" font-size="9">• Divergence (if/else) serializes — both paths execute</text>
  <text x="50" y="395" fill="#8b98a5" font-size="9">• Latency hidden by switching warps (not caching)</text>
  <text x="50" y="411" fill="#6b7684" font-size="8.5">Think in warps, not threads. Avoid divergence. Coalesce memory.</text>

  <!-- Programming model note -->
  <text x="380" y="435" fill="#e6edf3" font-size="9.5" text-anchor="middle" font-weight="600">CUDA = C++ with extensions: __global__ (kernel), __shared__ (fast memory), threadIdx/blockIdx (identity)</text>
  <text x="380" y="450" fill="#8b98a5" font-size="9" text-anchor="middle">PyTorch calls CUDA kernels underneath — every .to("cuda") triggers this execution model</text>

  <text x="380" y="464" fill="#6b7684" font-size="11" text-anchor="middle">CUDA is the API that turned GPUs from graphics into general compute — and locked the AI industry to NVIDIA.</text>
</svg>

Hierarchy:

Level        | Unit          | Maps To
-------------|---------------|-------------------
Grid         | Full workload | Kernel launch
Block        | Thread group  | Streaming Multiprocessor
Thread       | Single worker | CUDA core
Warp         | 32 threads    | Execution unit

Simple CUDA Example

Vector Addition:

// Kernel definition
__global__ void vectorAdd(float *a, float *b, float *c, int n) {
    // Calculate global thread ID
    int i = blockDim.x * blockIdx.x + threadIdx.x;
    
    if (i < n) {
        c[i] = a[i] + b[i];
    }
}

// Host code
int main() {
    int n = 1000000;
    float *d_a, *d_b, *d_c;
    
    // Allocate GPU memory
    cudaMalloc(&d_a, n * sizeof(float));
    cudaMalloc(&d_b, n * sizeof(float));
    cudaMalloc(&d_c, n * sizeof(float));
    
    // Copy data to GPU
    cudaMemcpy(d_a, h_a, n * sizeof(float), cudaMemcpyHostToDevice);
    cudaMemcpy(d_b, h_b, n * sizeof(float), cudaMemcpyHostToDevice);
    
    // Launch kernel
    int blockSize = 256;
    int numBlocks = (n + blockSize - 1) / blockSize;
    vectorAdd<<<numBlocks, blockSize>>>(d_a, d_b, d_c, n);
    
    // Copy result back
    cudaMemcpy(h_c, d_c, n * sizeof(float), cudaMemcpyDeviceToHost);
    
    // Free GPU memory
    cudaFree(d_a); cudaFree(d_b); cudaFree(d_c);
}

CUDA Libraries

Key Libraries:

Library      | Purpose
-------------|----------------------------------
cuDNN        | Deep learning primitives
cuBLAS       | Linear algebra (BLAS)
cuFFT        | Fast Fourier transforms
cuSPARSE     | Sparse matrix operations
cuRAND       | Random number generation
NCCL         | Multi-GPU communication
TensorRT     | Inference optimization

Framework Integration:

Framework    | CUDA Usage
-------------|----------------------------------
PyTorch      | torch.cuda, automatic dispatch
TensorFlow   | GPU ops, XLA compilation
JAX          | XLA with CUDA backend
RAPIDS       | GPU data science

PyTorch CUDA Usage

import torch

# Check CUDA availability
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.cuda.current_device())

# Move tensor to GPU
x = torch.randn(1000, 1000)
x_gpu = x.cuda()  # or x.to("cuda")

# Operations on GPU
y_gpu = x_gpu @ x_gpu.T  # Matrix multiply on GPU

# Move back to CPU
y_cpu = y_gpu.cpu()

# Specify device
device = torch.device("cuda:0")
model = MyModel().to(device)

CUDA Versions

CUDA Version | Features                    | Driver
-------------|-----------------------------|---------
12.x         | Hopper support, async       | 525+
11.x         | Ampere, BF16, TF32          | 450+
10.x         | Turing, mixed precision     | 410+

Version Checking:

# CUDA toolkit version
nvcc --version

# Driver version
nvidia-smi

# PyTorch CUDA version
python -c "import torch; print(torch.version.cuda)"

CUDA is the essential infrastructure of AI computing — while alternatives exist, CUDA's maturity, optimization, and ecosystem integration make it the de facto standard for AI development, with most frameworks, models, and workflows assuming CUDA-enabled NVIDIA GPUs.

cudanvidiaprogramminggpukernelparallelcudnn

Explore 500+ Semiconductor & AI Topics

From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.

🔍 Search Topics 💬 Ask CFSGPT 📚 Browse All

Related Topics

Explore 500+ Semiconductor & AI Topics