interconnect topology hpc
**HPC Interconnect Topologies** are the **physical and logical network structures that connect compute nodes in a supercomputer or data center cluster — where the choice of topology (fat tree, dragonfly, torus, mesh) determines the bisection bandwidth, diameter, cost, and scalability of the system, directly impacting the performance of communication-intensive parallel applications by 2-10x compared to a mismatched topology**.
**Why Topology Matters**
Parallel applications communicate through the interconnect — MPI collectives, distributed-memory data exchange, gradient synchronization in distributed training. The interconnect's bandwidth, latency, and congestion characteristics under real traffic patterns determine whether computation or communication is the bottleneck. A topology optimized for the workload's communication pattern can halve runtime.
**Key Topologies**
- **Fat Tree (Clos Network)**: A multi-level tree where bandwidth increases toward the root (hence "fat"). Every pair of nodes has full bisection bandwidth — any node can communicate with any other at line rate without congestion. The standard for data center and cloud clusters (used by almost all InfiniBand and Ethernet HPC installations). Drawback: many switches in the upper tiers (cost and power).
- **Dragonfly**: A hierarchical topology with three levels: routers within a group are fully connected; groups are connected by global links with each group reaching every other group in at most 2 hops. Provides near-full bisection bandwidth with fewer global cables than fat trees. Used in Cray Aries (Theta, Piz Daint) and Slingshot (Frontier). Requires adaptive routing to avoid congestion on global links.
- **3D Torus**: Each node is connected to its 6 nearest neighbors in a 3D grid with wrap-around links. Low radix (few cables per node), low cost, excellent for nearest-neighbor communication patterns (stencil computations, PDE solvers). Used in IBM Blue Gene and Fujitsu Fugaku (6D torus). Drawback: high diameter — worst-case communication traverses N^(1/3) hops.
- **Hypercube**: 2^n nodes, each connected to n neighbors differing in one bit of the node address. Diameter = n = log₂(N), excellent for global communication patterns. Impractical for large N due to high node degree, but the theoretical comparison baseline.
**Key Metrics**
| Metric | Definition | Impact |
|--------|-----------|--------|
| **Bisection Bandwidth** | Total bandwidth across a minimum cut dividing the network in half | Determines max all-to-all throughput |
| **Diameter** | Maximum hops between any two nodes | Determines worst-case latency |
| **Node Degree (Radix)** | Number of links per node/switch | Determines hardware cost per node |
| **Path Diversity** | Number of alternative paths between node pairs | Determines congestion resilience |
**Adaptive and Minimal Routing**
Modern interconnects use adaptive routing — dynamically selecting among multiple shortest-path alternatives based on real-time congestion information from switch buffers. Non-minimal (Valiant) routing sends packets through a random intermediate node, provably balancing load at the cost of doubling average hop count.
HPC Interconnect Topologies are **the circulatory system of parallel computing** — determining how fast data flows between the processors that form the parallel machine, and representing one of the most impactful architectural decisions in system design.