interconnect topology hpc

**HPC Interconnect Topologies** are the **network architectures that connect thousands to millions of compute nodes in supercomputers and data centers — where the choice of topology (fat tree, torus, dragonfly) determines the bisection bandwidth, latency, scalability, and cost that ultimately dictate whether the system can efficiently run communication-intensive parallel applications at scale**. **Why Topology Matters** A parallel application running on 10,000 nodes generates enormous inter-node communication (MPI collectives, parameter synchronization, halo exchanges). If the network topology creates bottlenecks where too many flows compete for the same links, application performance degrades catastrophically — even if every individual node has abundant compute power. **Major Topologies** - **Fat Tree (Clos Network)**: A hierarchical tree where bandwidth increases toward the root — upper-level switches have more ports or more links than lower levels, preventing the congestion that plagues simple trees. Non-blocking fat trees provide full bisection bandwidth (any half of the nodes can communicate with the other half at full link speed simultaneously). Used in most InfiniBand HPC clusters and Ethernet data centers. Advantages: well-understood routing, excellent worst-case performance. Disadvantages: expensive at scale (many switches and cables in upper tiers), high cabling complexity. - **3D/5D Torus**: Each node connects to its nearest neighbors in a 3D-6D grid, with wrap-around links forming a torus. Cray XC (Aries) and Fujitsu A64FX (Tofu) use torus topologies. Advantages: simple, regular structure; excellent for nearest-neighbor communication patterns (stencil codes, climate models); low switch count (switches are integrated into each node). Disadvantages: diameter grows as N^(1/d), so latency for distant nodes increases with system size; bisection bandwidth is lower than fat tree. - **Dragonfly**: A hierarchical design with three levels: nodes within a group are fully connected, and groups are connected by global links in a balanced pattern. HPE Slingshot and Cray Aries use dragonfly-like topologies. Advantages: low diameter (any-to-any in 3-4 hops), cost-effective (fewer global cables than fat tree), good for all-to-all communication. Disadvantages: adversarial traffic patterns can cause congestion on inter-group links; requires adaptive routing to balance load. - **Hypercube**: Each of N = 2^k nodes connects to k neighbors. Diameter = k = log2(N). Historically important but impractical at large scale due to the high per-node port count. **Performance Metrics** | Metric | Fat Tree | 3D Torus | Dragonfly | |--------|----------|----------|-----------| | **Diameter** | O(log N) | O(N^(1/3)) | O(1) (constant 3-4 hops) | | **Bisection BW** | Full | O(N^(2/3)) | Moderate | | **Switch Count** | High | Low | Moderate | | **Best For** | General-purpose | Nearest-neighbor | Mixed workloads | HPC Interconnect Topologies are **the highway system of supercomputing** — determining whether data flows freely between any two compute nodes or gets stuck in traffic jams that starve processors of the data they need to keep computing.