topology-aware training

**Topology-aware training** is the **distributed training placement strategy that maps communication-heavy ranks to favorable physical network paths** - it minimizes hop count and congestion by aligning algorithm communication patterns with cluster wiring. **What Is Topology-aware training?** - **Definition**: Rank assignment and process grouping that account for switch hierarchy, link speed, and locality. - **Communication Sensitivity**: All-reduce and tensor-parallel workloads are highly affected by physical placement. - **Placement Inputs**: Node adjacency, NIC affinity, NVLink topology, and rack-level oversubscription ratios. - **Output**: Lower collective latency, reduced cross-fabric traffic, and improved step-time stability. **Why Topology-aware training Matters** - **Performance**: Poor placement can erase expected scaling gains despite sufficient compute capacity. - **Network Efficiency**: Localizing heavy traffic reduces pressure on shared spine links. - **Cost**: Better topology use can delay expensive network upgrades. - **Reliability**: Less congestion reduces timeout and transient communication failures. - **Scalability**: Topology-aware mapping becomes critical as cluster size and job concurrency increase. **How It Is Used in Practice** - **Rank Mapping**: Place nearest-neighbor or frequent-communicating ranks on low-latency local paths. - **Scheduler Integration**: Expose network topology metadata to orchestration and placement logic. - **Feedback Loop**: Use profiler communication traces to refine placement heuristics over time. Topology-aware training is **a high-leverage systems optimization for large clusters** - matching logical communication to physical network reality materially improves distributed throughput.