all-reduce operation

**All-reduce operation** is the **collective communication primitive that aggregates values from all ranks and returns the result to each rank** - it is the core primitive used for gradient averaging in synchronous distributed training. **What Is All-reduce operation?** - **Definition**: Each worker contributes a tensor, reduction is applied, and reduced tensor is delivered to all workers. - **Common Reductions**: Sum and mean are most common for gradient synchronization and metric aggregation. - **Algorithm Families**: Ring, tree, and hybrid algorithms with different latency-bandwidth tradeoffs. - **Bottleneck Risk**: Inefficient all-reduce can limit scaling even when compute capacity is abundant. **Why All-reduce operation Matters** - **Distributed Correctness**: Ensures all workers share a consistent global gradient view. - **Throughput Impact**: Collective latency directly enters step time at large cluster scale. - **Topology Sensitivity**: Choosing the right algorithm for network structure improves efficiency materially. - **Framework Foundation**: Most distributed libraries rely on all-reduce as the default synchronization path. - **Optimization Leverage**: All-reduce tuning often yields immediate measurable speed gains. **How It Is Used in Practice** - **Bucket Sizing**: Tune gradient bucket sizes to balance launch overhead and overlap opportunities. - **Algorithm Selection**: Use ring for bandwidth-bound regimes and trees for latency-sensitive cases. - **Fabric Validation**: Benchmark all-reduce bandwidth and tail latency under realistic cluster load. All-reduce operation is **the primary communication kernel of synchronous distributed learning** - its efficiency largely determines practical scaling limits for data-parallel training.