performance profiling parallel

**Performance Profiling** is the **measurement and analysis of where a parallel program spends time and resources** — identifying bottlenecks that limit performance and guiding optimization efforts to maximum effect. **Profiling Workflow** 1. **Hypothesis**: Where is the bottleneck? (CPU compute? Memory? GPU kernel? Communication?) 2. **Instrument**: Enable profiling — minimal overhead tools preferred. 3. **Collect**: Run with profiler attached → gather data. 4. **Analyze**: Identify top time consumers, hotspots, stalls. 5. **Optimize**: Fix bottleneck. 6. **Verify**: Measure speedup, ensure no regression. **GPU Profiling Tools** **NVIDIA Nsight Systems**: - System-wide timeline: CPU threads, CUDA kernels, memory transfers, NVLink. - Shows GPU utilization, transfer overlap, synchronization gaps. - CLI: `nsys profile --trace=cuda,nvtx ./app` **NVIDIA Nsight Compute**: - Kernel-level analysis: Throughput, occupancy, instruction mix, memory bandwidth. - Roofline model view: Is kernel compute-bound or memory-bound? - Source-level metrics: Which lines have most cache misses. **CPU Profiling Tools** **Intel VTune**: - Hotspot analysis, threading, memory access patterns. - Microarchitecture analysis: Front-end stalls, back-end stalls, cache misses. - Platform: Windows and Linux, all Intel and AMD CPUs. **Linux perf**: - Sampling profiler: `perf record -g ./app` then `perf report`. - Hardware counters: cache-misses, branch-misses, cycles, instructions. - Flame graphs: Hierarchical call-stack visualization. **Memory Profiling** - Valgrind Massif: Heap memory usage over time. - CUDA memcheck / compute-sanitizer: GPU memory errors. - Heaptrack: Fast heap profiler with stack unwinding. **Key Metrics to Examine** - **GPU**: SM utilization, memory bandwidth utilization, occupancy, warp efficiency. - **CPU**: Instructions per cycle (IPC), cache miss rate, vectorization ratio. - **MPI**: Communication time fraction, synchronization overhead, load imbalance. **Amdahl's Law in Practice** - Profile first: Optimize the 20% of code that takes 80% of time. - Common mistake: Optimize clean code that contributes < 1% of runtime. Performance profiling is **the scientific method for parallel optimization** — without measurement, optimization is guesswork; with proper profiling, optimization effort can be directed to where it matters most, achieving maximum speedup per engineering hour invested.