distributed training hierarchical allreduce

**Hierarchical all-reduce** is the **two-level collective strategy that reduces gradients within nodes first, then across nodes** - it exploits faster intra-node links and minimizes traffic on slower inter-node network paths. **What Is Hierarchical all-reduce?** - **Definition**: Perform local reduction among GPUs in a node, then global reduction among node representatives. - **Topology Fit**: Designed for systems with high intra-node bandwidth such as NVLink and slower cross-node fabric. - **Communication Pattern**: Reduces volume and contention on inter-node links compared with flat collectives. - **Implementation**: Often provided via optimized NCCL or framework-level collective selection policies. **Why Hierarchical all-reduce Matters** - **Scale Efficiency**: Improves step time at high node counts where network hierarchy is significant. - **Bandwidth Protection**: Limits pressure on expensive shared network tiers. - **Predictable Performance**: More stable collective latency under mixed workloads and large job counts. - **Cost-Performance**: Extracts better throughput from existing fabric without immediate hardware upgrades. - **Topology Utilization**: Turns hardware locality into measurable distributed-training speedup. **How It Is Used in Practice** - **Rank Mapping**: Place ranks to maximize local reductions on fastest links before cross-node phase. - **Collective Policy**: Enable hierarchical algorithm selection for large tensor reductions. - **Validation**: Compare flat versus hierarchical collectives across job sizes to choose break-even points. Hierarchical all-reduce is **a high-impact topology-aware communication optimization** - local-first reduction reduces network pressure and improves large-cluster training efficiency.