gpu performance profiling nsight

**GPU Performance Profiling** encompasses **systematic measurement and analysis of kernel execution, memory access patterns, and hardware utilization using Nsight tools, roofline models, and application-specific metrics to identify bottlenecks and guide optimization.** **Nsight Compute and Nsight Systems Overview** - **Nsight Compute**: Kernel-centric profiler. Analyzes single kernel execution: register/shared memory usage, L1/L2 cache hit rates, warp stall reasons, SM efficiency. - **Nsight Systems**: System-wide profiler. Timeline view of entire application: kernel launches, memory transfers, CPU-GPU synchronization, context switches, power consumption. - **Guided Analysis Workflow**: Nsight Compute recommends optimizations based on measured metrics (e.g., "warp occupancy 50%, increase shared memory usage to 75%"). - **Overhead**: Profiling adds ~5-50% runtime overhead depending on metric set. Light profiling (SM efficiency) minimal; heavy profiling (register spills) substantial. **NVTX Annotations for Custom Metrics** - **NVTX (NVIDIA Tools Extension)**: API to annotate application code. Marks user-defined ranges, domains, events with custom names. - **Range Annotation**: nvtxRangePush/Pop() delineate code sections. Nsight timeline shows annotated regions, enabling user-level performance tracking. - **Domain Separation**: nvtxDomainCreate() organizes related annotations. Example: separate domains for preprocessing, compute, postprocessing. - **Color and Category**: Annotations assigned colors (visual grouping) and categories (filtering). Facilitates timeline analysis of complex multi-threaded applications. **Roofline Model for GPU Analysis** - **Roofline Concept**: 2D plot of achievable GFLOP/s vs arithmetic intensity (FLOP per byte transferred). Machine peak provides "roofline" ceiling. - **Peak Compute Roofline**: GPU compute peak (theoretical FP32 FLOP/s). Ampere A100: 312 TFLOP/s peak. - **Peak Bandwidth Roofline**: GPU memory bandwidth (theoretical throughput). A100 HBM2e: 2 TB/s peak. Roofline ceiling = MIN(peak_compute, intensity × peak_bandwidth). - **Application Characterization**: Measure kernel arithmetic intensity (FLOP count / memory bytes transferred). Points below roofline indicate under-utilization. **Achieved Occupancy and Bottleneck Analysis** - **Occupancy Metric**: Percentage of SM warp slots filled. Occupancy = (resident_warps / max_warps_per_sm) × 100%. Max warp/SM: 64 (Volta), 48 (Ampere). - **Limiting Factors**: Register pressure (32k limit per SM), shared memory allocation (96KB per SM), thread blocks per SM (varies by GPU). - **Occupancy vs Performance**: Higher occupancy generally improves performance (more warps hide memory latency), but not always. Some high-register kernels benefit from lower occupancy. - **Warp Stall Reasons**: Nsight reports stall causes (memory, dependency, execution resource, synchronization). Prioritize fixing most-common stall. **Memory Bandwidth Utilization** - **Effective Bandwidth**: Measured memory bytes (profiler) vs theoretical peak. Typical ratios: 50-90% depending on access pattern. - **Coalescing Efficiency**: Consecutive threads accessing consecutive memory addresses coalesce into single transaction. Scattered access wastes bandwidth (cache-only reuse). - **Bank Conflicts**: Shared memory bank conflicts serialize accesses. All 32 threads accessing same bank → 32x slowdown. Proper access pattern avoids conflicts. - **L2 Cache Effectiveness**: L2 cache hit rate impacts bandwidth. Reuse distance (iterations between data access) determines cache utility. **Cache Utilization and Patterns** - **L1 Cache**: Per-SM cache (32-96KB depending on config). Caches load/store operations if enabled. Bank conflicts similar to shared memory. - **L2 Cache**: Shared across all SMs (4-40 MB depending on GPU). Victim cache for L1, also receives uncached loads. - **Hit Rate Interpretation**: High L1 hit rate (>80%) indicates locality; low ratio indicates poor spatial/temporal locality. - **Profiler L2 Analysis**: Misses per 1k instructions metric. Aim for <2-5 misses/1k instructions for well-optimized kernels. **SM Efficiency and Load Balancing** - **SM Efficiency**: Percentage of SM slots executing useful instructions. Idle slots due to warp stalls, divergence, or under-occupancy. - **Warp Divergence Analysis**: Branch divergence metrics show divergence frequency and impact. Serialization within warp reduces throughput. - **Grid-Level Load Balancing**: Blocks distributed unevenly → some SMs idle while others compute. Profiler shows block-per-SM histogram. - **Dynamic Parallelism Overhead**: Child kernels launched from kernel require synchronization overhead. Impacts SM efficiency if child kernels small. **Optimization Workflows** - **Memory-Bound Analysis**: If roofline point below bandwidth line, kernel memory-bound. Optimize: improve coalescing, increase data reuse, prefetching. - **Compute-Bound Analysis**: If roofline point below compute line, kernel compute-bound. Optimize: reduce instruction count, use tensor cores, improve ILP. - **Iterative Refinement**: Profile → identify bottleneck → optimize → re-profile. Typical 5-10 iteration cycle for 2-5x speedup.