performance profiling parallel
**Parallel Performance Profiling** is the **measurement and analysis discipline that identifies performance bottlenecks in parallel applications — pinpointing whether a program is limited by computation, memory bandwidth, communication, synchronization, or load imbalance, and quantifying the impact of each bottleneck using hardware performance counters, tracing, and statistical sampling to guide optimization toward the highest-impact changes**.
**Why Profiling Parallel Code Is Different**
Sequential profiling asks "which function is slowest?" Parallel profiling asks fundamentally different questions: "Why isn't this scaling to N cores?" "Which threads are waiting, and for what?" "Is the bottleneck computation, communication, or synchronization?" "What is the critical path?" Sequential hotspot analysis can be misleading in parallel code — the hottest function might be perfectly parallel while the actual bottleneck is a serialized lock.
**Profiling Methodologies**
- **Sampling (Statistical)**: Periodically interrupt each thread and record the program counter and call stack. After millions of samples, the function-level profile converges to the true time distribution. Low overhead (<5%). Tools: Intel VTune, Linux perf, AMD uProf.
- **Instrumentation (Tracing)**: Insert timestamps at every function entry/exit, MPI call, synchronization event. Produces a complete timeline of all threads' activities. High overhead (10-50%) but provides exact event ordering. Tools: Score-P, TAU, Vampir, NVIDIA Nsight Systems.
- **Hardware Performance Counters**: CPU/GPU hardware counts events: cache misses, branch mispredictions, instructions retired, memory bandwidth consumed, FLOPS executed. Counters quantify architectural bottlenecks without modifying the code. Tools: PAPI, likwid, VTune, Nsight Compute.
**Key Parallel Metrics**
| Metric | What It Reveals |
|--------|----------------|
| **Parallel Efficiency** | Speedup/P — how well P cores are utilized |
| **Load Imbalance** | max(thread_time)/avg(thread_time) — 1.0 is perfect |
| **Communication Time** | % of time in MPI/NCCL calls — communication overhead |
| **Synchronization Wait** | Time spent in barriers, locks, condition variables |
| **Memory Bandwidth Utilization** | Achieved vs. peak — memory-bound detection |
| **IPC (Instructions Per Cycle)** | Low IPC + high cache misses = memory-bound |
**GPU-Specific Profiling**
- **NVIDIA Nsight Compute**: Kernel-level profiling. Reports achieved occupancy, memory throughput, compute throughput, warp stall reasons, and roofline position for each kernel launch. The definitive tool for CUDA kernel optimization.
- **NVIDIA Nsight Systems**: System-level timeline showing CPU activity, GPU kernel launches, memory transfers, and CUDA API calls. Identifies gaps between kernel launches and CPU-GPU synchronization overhead.
**Scalability Analysis**
Profile at multiple scales (1, 2, 4, 8, 16, ... P) and plot speedup vs. P. Strong scaling (fixed total problem) reveals communication and synchronization overhead. Weak scaling (fixed per-processor problem) reveals algorithmic overhead. Deviation from linear scaling at specific P values pinpoints the bottleneck.
**Parallel Performance Profiling is the scientific method applied to optimization** — replacing guesswork with measurement-driven analysis that identifies the true limiting factor, ensuring that engineering effort is directed at the bottleneck that actually matters.