load balancing parallel computing

**Load Balancing in Parallel Computing** is **the process of distributing computational work evenly across all available processing units to minimize idle time and maximize throughput — directly determining the gap between theoretical linear speedup and actual achieved performance in parallel applications**. **Static Load Balancing:** - **Block Partitioning**: divide N work items into P equal blocks of N/P each — simple but assumes uniform cost per item; effective only when computation per item is identical and predictable - **Cyclic Partitioning**: assign items in round-robin fashion (item i to processor i mod P) — better than block when cost varies smoothly across items (e.g., triangular matrix operations where work decreases with row index) - **Block-Cyclic Partitioning**: combine block and cyclic by assigning blocks of B items cyclically — balances locality (block) with load distribution (cyclic); used in ScaLAPACK for dense linear algebra - **Graph Partitioning**: for irregular computations (mesh-based simulations, graph analytics), partition the computational graph into P balanced subsets with minimized edge cuts — METIS and ParMETIS are standard tools achieving <5% load imbalance **Dynamic Load Balancing:** - **Work Queue**: centralized queue distributes work items to processors on demand — each processor pulls next item when idle; granularity of work items controls overhead vs. balance tradeoff - **Work Stealing**: each processor has a local deque; idle processors steal from the bottom of a victim's deque — achieves provably near-optimal load balance with O(P × Tinfinity) total steal operations - **Task Splitting**: when a processor exhausts its work and no more is available, overloaded processors split their remaining work and share — enables dynamic rebalancing mid-computation without centralized coordination - **Guided Self-Scheduling**: remaining iterations divided by P and assigned as decreasing-size chunks — first chunks are large (good locality), later chunks are small (good balance); implemented in OpenMP schedule(guided) **Measuring and Diagnosing Imbalance:** - **Load Imbalance Factor**: max_time / average_time across processors — value of 1.0 is perfect balance; typical target <1.1 (less than 10% imbalance) - **Barrier Wait Time**: time processors spend waiting at barriers indicates imbalance — profiling tools (Intel VTune, NVIDIA Nsight Systems) show per-thread barrier wait time - **Application-Specific Metrics**: for iterative solvers, per-rank iteration time variance indicates work distribution quality — adaptive repartitioning triggered when variance exceeds threshold **Load balancing is the practical linchpin of parallel performance — Amdahl's Law describes the theoretical limit from serial fraction, but in practice load imbalance is equally devastating, as the slowest processor determines overall completion time regardless of how fast all other processors finish.**