warp

A warp (NVIDIA terminology) or wavefront (AMD) is a group of threads that execute together in lockstep on GPU hardware, typically 32 threads for NVIDIA and 64 for AMD, representing the fundamental unit of SIMT (Single Instruction Multiple Thread) execution. SIMT execution: all threads in warp execute same instruction simultaneously but on different data (like SIMD, but each thread has own registers and can diverge). Warp scheduling: GPU schedules warps, not individual threads; when one warp stalls (memory access), scheduler switches to ready warp—latency hiding. Thread divergence: if threads in warp take different branches (if-else), both paths execute serially with threads masked out; significant performance impact. Occupancy: number of active warps per SM divided by maximum; higher occupancy generally helps hide latency. Warp-level primitives: special operations across warp threads—__shfl (shuffle data between threads), __ballot (vote), and __reduce (reduction). Memory coalescing: threads in warp should access adjacent memory for efficient memory transactions. Vectorization: warps effectively vectorize operations; design kernels to maximize utilization. Register pressure: each thread needs registers; more registers per thread means fewer concurrent warps. Performance optimization: minimize divergence, maximize coalescing, and tune occupancy. Understanding warps is essential for GPU programming and performance optimization.