GPU (Graphics Processing Unit) is a specialized processor designed for parallel processing tasks

• GPUs: Plural form of GPU
• Graphics Card: Physical hardware component containing a GPU, VRAM, and cooling system
• Accelerator: Specialized hardware that offloads computation from the CPU

Architecture Fundamentals

Core Components

• Streaming Multiprocessors (SMs): Contain multiple CUDA cores for parallel execution
• VRAM (Video RAM): High-bandwidth memory dedicated to the GPU
• Memory Bus: Data pathway between GPU and VRAM
• PCIe Interface: Connection to the motherboard/CPU

Parallelism Model

GPUs excel at SIMD (Single Instruction, Multiple Data) operations:

$$ \text{Speedup} = \frac{T_{\text{sequential}}}{T_{\text{parallel}}} \leq \frac{1}{(1-P) + \frac{P}{N}} $$

Where:

• $P$ = Parallelizable fraction of code
• $N$ = Number of parallel processors
• This is Amdahl's Law

Performance Metrics

FLOPS (Floating Point Operations Per Second)

$$ \text{FLOPS} = \text{Cores} \times \text{Clock Speed (Hz)} \times \text{FLOPs per cycle} $$

Example calculation for a GPU with 10,000 cores at 2 GHz:

$$ \text{FLOPS} = 10{,}000 \times 2 \times 10^9 \times 2 = 40 \text{ TFLOPS} $$

Memory Bandwidth

$$ \text{Bandwidth (GB/s)} = \frac{\text{Memory Clock (Hz)} \times \text{Bus Width (bits)} \times \text{Data Rate}}{8 \times 10^9} $$

Arithmetic Intensity

$$ \text{Arithmetic Intensity} = \frac{\text{FLOPs}}{\text{Bytes Accessed}} $$

The Roofline Model bounds performance:

$$ \text{Attainable FLOPS} = \min\left(\text{Peak FLOPS}, \text{Bandwidth} \times \text{Arithmetic Intensity}\right) $$

GPU Computing Concepts

Thread Hierarchy (CUDA Model)

• Thread: Smallest unit of execution
• Each thread has unique indices: threadIdx.x, threadIdx.y, threadIdx.z
• Block: Group of threads that can cooperate
• Shared memory accessible within block
• Maximum threads per block: typically 1024
• Grid: Collection of blocks
• Total threads: $\text{Grid Size} \times \text{Block Size}$

Memory Hierarchy

Matrix Operations (Key for AI/ML)

Matrix Multiplication Complexity

Standard matrix multiplication for $A_{m \times k} \cdot B_{k \times n}$:

$$ C_{ij} = \sum_{l=1}^{k} A_{il} \cdot B_{lj} $$

• Time Complexity: $O(m \times n \times k)$
• Naive: $O(n^3)$ for square matrices
• Strassen's Algorithm: $O(n^{2.807})$

Tensor Core Operations

Mixed-precision matrix multiply-accumulate:

$$ D = A \times B + C $$

Where:

• $A, B$ are FP16 (16-bit floating point)
• $C, D$ are FP32 (32-bit floating point)

Throughput comparison:

• FP32 CUDA Cores: ~40 TFLOPS
• FP16 Tensor Cores: ~300+ TFLOPS
• INT8 Tensor Cores: ~600+ TFLOPS

Power and Thermal Equations

Thermal Design Power (TDP)

$$ P_{\text{dynamic}} = \alpha \cdot C \cdot V^2 \cdot f $$

Where:

• $\alpha$ = Activity factor
• $C$ = Capacitance
• $V$ = Voltage
• $f$ = Frequency

Temperature Relationship

$$ T_{\text{junction}} = T_{\text{ambient}} + (P \times R_{\theta}) $$

Where $R_{\theta}$ is thermal resistance in °C/W.

Deep Learning Operations

Convolution (CNN)

For a 2D convolution with input $I$, kernel $K$, output $O$:

$$ O(i,j) = \sum_{m}\sum_{n} I(i+m, j+n) \cdot K(m,n) $$

Output dimensions:

$$ O_{\text{size}} = \left\lfloor \frac{I_{\text{size}} - K_{\text{size}} + 2P}{S} \right\rfloor + 1 $$

Where:

• $P$ = Padding
• $S$ = Stride

Attention Mechanism (Transformers)

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Memory complexity: $O(n^2 \cdot d)$ where $n$ is sequence length.

Major GPU Vendors

NVIDIA

• Gaming: GeForce RTX series
• Professional: Quadro / RTX A-series
• Data Center: A100, H100, H200, B100, B200
• CUDA Ecosystem: Dominant in AI/ML

AMD

• Gaming: Radeon RX series
• Data Center: Instinct MI series (MI300X)
• ROCm: Open-source GPU computing platform

Intel

• Consumer: Arc A-series
• Data Center: Gaudi accelerators, Max series

Code Example: CUDA Kernel

// Vector addition kernel
__global__ void vectorAdd(float *A, float *B, float *C, int N) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < N) {
        C[idx] = A[idx] + B[idx];
    }
}

// Launch configuration
int threadsPerBlock = 256;
int blocksPerGrid = (N + threadsPerBlock - 1) / threadsPerBlock;
vectorAdd<<<blocksPerGrid, threadsPerBlock>>>(d_A, d_B, d_C, N);

Quick Reference Formulas

References

• NVIDIA CUDA Programming Guide
• AMD ROCm Documentation
• Patterson & Hennessy, Computer Architecture