GPU Performance Profiling encompasses systematic measurement and analysis of kernel execution, memory access patterns, and hardware utilization using Nsight tools, roofline models, and application-specific metrics to identify bottlenecks and guide optimization.
Nsight Compute and Nsight Systems Overview
- Nsight Compute: Kernel-centric profiler. Analyzes single kernel execution: register/shared memory usage, L1/L2 cache hit rates, warp stall reasons, SM efficiency.
- Nsight Systems: System-wide profiler. Timeline view of entire application: kernel launches, memory transfers, CPU-GPU synchronization, context switches, power consumption.
- Guided Analysis Workflow: Nsight Compute recommends optimizations based on measured metrics (e.g., "warp occupancy 50%, increase shared memory usage to 75%").
- Overhead: Profiling adds ~5-50% runtime overhead depending on metric set. Light profiling (SM efficiency) minimal; heavy profiling (register spills) substantial.
NVTX Annotations for Custom Metrics
- NVTX (NVIDIA Tools Extension): API to annotate application code. Marks user-defined ranges, domains, events with custom names.
- Range Annotation: nvtxRangePush/Pop() delineate code sections. Nsight timeline shows annotated regions, enabling user-level performance tracking.
- Domain Separation: nvtxDomainCreate() organizes related annotations. Example: separate domains for preprocessing, compute, postprocessing.
- Color and Category: Annotations assigned colors (visual grouping) and categories (filtering). Facilitates timeline analysis of complex multi-threaded applications.
Roofline Model for GPU Analysis
- Roofline Concept: 2D plot of achievable GFLOP/s vs arithmetic intensity (FLOP per byte transferred). Machine peak provides "roofline" ceiling.
- Peak Compute Roofline: GPU compute peak (theoretical FP32 FLOP/s). Ampere A100: 312 TFLOP/s peak.
- Peak Bandwidth Roofline: GPU memory bandwidth (theoretical throughput). A100 HBM2e: 2 TB/s peak. Roofline ceiling = MIN(peak_compute, intensity × peak_bandwidth).
- Application Characterization: Measure kernel arithmetic intensity (FLOP count / memory bytes transferred). Points below roofline indicate under-utilization.
Achieved Occupancy and Bottleneck Analysis
- Occupancy Metric: Percentage of SM warp slots filled. Occupancy = (resident_warps / max_warps_per_sm) × 100%. Max warp/SM: 64 (Volta), 48 (Ampere).
- Limiting Factors: Register pressure (32k limit per SM), shared memory allocation (96KB per SM), thread blocks per SM (varies by GPU).
- Occupancy vs Performance: Higher occupancy generally improves performance (more warps hide memory latency), but not always. Some high-register kernels benefit from lower occupancy.
- Warp Stall Reasons: Nsight reports stall causes (memory, dependency, execution resource, synchronization). Prioritize fixing most-common stall.
Memory Bandwidth Utilization
- Effective Bandwidth: Measured memory bytes (profiler) vs theoretical peak. Typical ratios: 50-90% depending on access pattern.
- Coalescing Efficiency: Consecutive threads accessing consecutive memory addresses coalesce into single transaction. Scattered access wastes bandwidth (cache-only reuse).
- Bank Conflicts: Shared memory bank conflicts serialize accesses. All 32 threads accessing same bank → 32x slowdown. Proper access pattern avoids conflicts.
- L2 Cache Effectiveness: L2 cache hit rate impacts bandwidth. Reuse distance (iterations between data access) determines cache utility.
Cache Utilization and Patterns
- L1 Cache: Per-SM cache (32-96KB depending on config). Caches load/store operations if enabled. Bank conflicts similar to shared memory.
- L2 Cache: Shared across all SMs (4-40 MB depending on GPU). Victim cache for L1, also receives uncached loads.
- Hit Rate Interpretation: High L1 hit rate (>80%) indicates locality; low ratio indicates poor spatial/temporal locality.
- Profiler L2 Analysis: Misses per 1k instructions metric. Aim for <2-5 misses/1k instructions for well-optimized kernels.
SM Efficiency and Load Balancing
- SM Efficiency: Percentage of SM slots executing useful instructions. Idle slots due to warp stalls, divergence, or under-occupancy.
- Warp Divergence Analysis: Branch divergence metrics show divergence frequency and impact. Serialization within warp reduces throughput.
- Grid-Level Load Balancing: Blocks distributed unevenly → some SMs idle while others compute. Profiler shows block-per-SM histogram.
- Dynamic Parallelism Overhead: Child kernels launched from kernel require synchronization overhead. Impacts SM efficiency if child kernels small.
Optimization Workflows
- Memory-Bound Analysis: If roofline point below bandwidth line, kernel memory-bound. Optimize: improve coalescing, increase data reuse, prefetching.
- Compute-Bound Analysis: If roofline point below compute line, kernel compute-bound. Optimize: reduce instruction count, use tensor cores, improve ILP.
- Iterative Refinement: Profile → identify bottleneck → optimize → re-profile. Typical 5-10 iteration cycle for 2-5x speedup.