gpu profiling optimization

**GPU Performance Profiling and Optimization** is the **systematic analysis methodology that identifies and eliminates performance bottlenecks in GPU kernels — using hardware performance counters, execution traces, and roofline analysis to determine whether a kernel is limited by compute throughput, memory bandwidth, latency, or occupancy, then applying targeted optimizations that can improve kernel performance by 2-10x compared to a naive implementation**. **Profiling Tools** - **NVIDIA Nsight Compute**: Kernel-level profiler that collects hundreds of hardware metrics per kernel launch. Reports achieved throughput vs. peak (compute utilization, memory throughput), warp execution efficiency, register usage, shared memory usage, and detailed pipeline stall reasons. - **NVIDIA Nsight Systems**: System-level profiler showing the timeline of GPU kernel launches, memory transfers, CPU activity, and API calls. Identifies gaps where the GPU is idle (kernel launch latency, synchronization waits, host-device transfer bottlenecks). - **AMD ROCprofiler / Omniperf**: Equivalent profiling tools for AMD GPUs, providing similar hardware counter access and roofline analysis. **Key Performance Metrics** - **SM Occupancy**: Ratio of active warps to maximum warps per SM. Higher occupancy helps hide memory latency (more warps to switch to while waiting for memory). Limited by register usage, shared memory usage, and block size. - **Compute Throughput**: % of peak FLOPS achieved. Low compute throughput on a compute-bound kernel indicates instruction-level inefficiencies (poor ILP, warp divergence). - **Memory Throughput**: % of peak memory bandwidth achieved. Low memory throughput on a memory-bound kernel indicates uncoalesced accesses, bank conflicts, or insufficient in-flight memory requests. - **Warp Execution Efficiency**: % of active lanes per warp instruction. Below 100% indicates branch divergence — threads within a warp taking different paths. **Common Bottlenecks and Optimizations** - **Uncoalesced Memory Access**: Adjacent threads access non-adjacent global memory addresses, causing multiple memory transactions instead of one. Fix: restructure data layout (AoS → SoA — Array of Structures to Structure of Arrays). - **Shared Memory Bank Conflicts**: Multiple threads in a warp access the same shared memory bank simultaneously. Fix: pad shared memory arrays (add one element per row) to shift access patterns across banks. - **Low Occupancy**: Kernel uses too many registers (>64 per thread), limiting the number of concurrent warps. Fix: reduce register pressure by simplifying per-thread computation or using launch_bounds to hint the compiler. - **Kernel Launch Overhead**: Many small kernels create a stream of short launches with GPU idle gaps between them. Fix: fuse kernels, use CUDA graphs to batch launches, or increase per-kernel work. - **Branch Divergence**: Conditional branches cause warp serialization. Fix: restructure computation so all threads in a warp take the same path, or reorganize data so divergent work is across warps rather than within. **The Optimization Cycle** 1. Profile → identify the bottleneck (compute? memory? latency?). 2. Optimize the identified bottleneck. 3. Re-profile → verify improvement and identify the new bottleneck. 4. Repeat until reaching the roofline ceiling. GPU Profiling is **the empirical science of GPU performance** — because intuition about where bottlenecks lie is almost always wrong in the complex, highly parallel execution environment of a modern GPU, and only measurement-driven optimization reliably delivers the performance gains that justify the GPU's hardware investment.