parallelism

SIMD (Single Instruction Multiple Data) and SIMT (Single Instruction Multiple Thread) are parallel execution models where GPUs excel, enabling the massive parallelism required for matrix operations and deep learning workloads. SIMD: single instruction operates on multiple data elements simultaneously in vector registers; CPU vector extensions (SSE, AVX, NEON) implement SIMD. SIMT: GPU model where a single instruction executes across many threads; each thread has own registers but shares instruction stream. GPU advantage: thousands of cores executing SIMT; optimized for data-parallel workloads where same operation applied to many elements. Matrix operations: matrix multiplication is inherently parallel—each output element computed independently; SIMD/SIMT provides massive speedup. Warp/Wavefront: SIMT execution groups (32 threads for NVIDIA, 64 for AMD) that execute together. Divergence: when threads take different branches, SIMT serializes paths; minimize branching in GPU code. Memory coalescing: adjacent threads should access adjacent memory for efficient SIMT execution. Vectorization: compilers auto-vectorize loops for SIMD; explicit intrinsics for fine control. Deep learning: matrix multiplications dominate training and inference; GPU SIMT provides 10-100× speedup over CPU. Tensor Cores: specialized matrix units extend beyond basic SIMT for AI workloads. Understanding SIMD/SIMT is fundamental for optimizing parallel computations.