gpu kernel optimization
**GPU Kernel Optimization** is the **systematic process of tuning GPU compute kernels to maximize hardware utilization and minimize execution time**, addressing memory access patterns, occupancy, instruction mix, and resource allocation to approach the theoretical peak performance defined by the roofline model.
GPU performance optimization follows a hierarchy: first ensure the algorithm is appropriate for GPU execution (sufficient parallelism, minimal branching), then optimize memory access patterns, then tune occupancy and resource usage, and finally optimize instruction-level details.
**Memory Optimization** (usually the biggest impact):
| Pattern | Problem | Solution |
|---------|---------|----------|
| Uncoalesced global loads | Bandwidth waste | Restructure data layout (AoS to SoA) |
| Bank conflicts in shared mem | Serialization | Pad shared memory arrays |
| Register spilling | Slow local memory access | Reduce register pressure per thread |
| Redundant global loads | Wasted bandwidth | Cache in shared memory or registers |
| Unaligned access | Extra transactions | Align data to 128-byte boundaries |
**Occupancy Tuning**: Occupancy = active warps / maximum warps per SM. Higher occupancy hides memory latency through warp switching. Occupancy is limited by: **registers per thread** (more registers mean fewer warps fit), **shared memory per block** (more shared memory means fewer blocks per SM), and **threads per block** (must be multiple of warp size). Use CUDA occupancy calculator or launch bounds to find optimal balance.
However, **maximum occupancy is not always optimal**: some kernels perform better at lower occupancy because: more registers per thread eliminate spilling, more shared memory per block enables larger tiles, and fewer active warps reduce cache thrashing. Profile-guided optimization is essential.
**Instruction-Level Optimization**: **Minimize expensive operations** (division, modulo — use bitwise for powers of 2; transcendentals — use fast-math intrinsics); **use intrinsics** (warp shuffle, ballot, popcount for collective operations); **loop unrolling** (reduces branch overhead, enables instruction-level parallelism); **predication** for short branches (avoid warp divergence); and **fused multiply-add** (FMA provides 2 FLOPs per instruction).
**Launch Configuration Optimization**: **Grid/block dimensions** affect both occupancy and memory access patterns. Block size should be a multiple of 32 (warp size); 128 or 256 threads per block is a common starting point. Grid size should provide enough blocks to fill all SMs (at least 2x number of SMs for load balancing). For workloads with variable execution time per thread, use persistent-thread or thread-block-cluster approaches.
**Profiling-Driven Workflow**: Use NVIDIA Nsight Compute (NCU) or AMD ROCProfiler to identify bottlenecks: **memory-bound** (low compute utilization, high memory utilization — optimize accesses), **compute-bound** (high compute utilization — optimize instructions or increase parallelism), **latency-bound** (low utilization for both — increase occupancy or reduce dependencies).
**GPU kernel optimization is an empirical discipline where theoretical analysis guides initial design but profiling-driven iteration delivers final performance — the gap between a naive and optimized kernel can be 10-100x, making optimization expertise one of the highest-leverage skills in GPU computing.**