interconnect topology
**Interconnect Topology** is the **physical and logical arrangement of network links connecting compute nodes in parallel systems** — determining the bandwidth, latency, scalability, and cost characteristics of the communication fabric that enables thousands to millions of processors to work together, with topology choice directly impacting application performance by 2-5x for communication-heavy workloads.
**Common Topologies**
| Topology | Bisection BW | Diameter | Cost (links) | Used By |
|----------|-------------|---------|-------------|--------|
| Fat Tree | Full bisection | 2 log N | O(N log N) | Ethernet clusters, InfiniBand |
| 3D Torus | O(N^(2/3)) | O(N^(1/3)) | O(N) | IBM Blue Gene, Fugaku |
| Dragonfly | ~Full bisection | 3-5 hops | O(N^(4/3)) | Cray XC/Slingshot |
| Hypercube | O(N) | log N | O(N log N) | Historical (CM-2) |
| Mesh (2D/3D) | O(√N) | O(√N) | O(N) | GPU NVSwitch mesh |
**Fat Tree (Clos Network)**
- **Structure**: Multi-level tree with increasing bandwidth at each level → "fat" at top.
- **Full bisection bandwidth**: Any half of nodes can communicate with other half at full speed.
- **Implementation**: Standard Ethernet/InfiniBand switches in leaf-spine-core layers.
- **Pros**: Non-blocking, any-to-any communication at full bandwidth.
- **Cons**: Expensive — top-level switches carry all traffic. Cable count: O(N log N).
**3D Torus**
- **Structure**: Each node connected to 6 neighbors (±x, ±y, ±z). Wrap-around links at edges.
- **IBM Blue Gene/Q**: 5D torus with 10 links per node.
- **Fujitsu Fugaku (#1 in 2020)**: 6D mesh/torus (Tofu-D interconnect).
- **Pros**: Simple, low cost (O(N) links), good for nearest-neighbor communication (stencil patterns).
- **Cons**: Low bisection bandwidth — all-to-all communication suffers.
**Dragonfly**
- **Structure**: Three levels — intra-group (local), inter-group (global), inter-cabinet.
- **Groups**: Fully connected internally. Groups connected by global links.
- **Adaptive routing**: Traffic dynamically routed to avoid congestion.
- **Cray Slingshot / HPE Cray EX**: Modern HPC systems use Dragonfly variants.
- **Pros**: Good balance of cost, bandwidth, and latency for diverse traffic patterns.
**NVLink/NVSwitch Topology (GPU clusters)**
- **DGX A100 (8 GPUs)**: Full NVSwitch mesh — any GPU to any GPU at 600 GB/s.
- **DGX H100 (8 GPUs)**: NVSwitch 4th gen — 900 GB/s per GPU.
- **NVLink Network (multi-node)**: NVLink extended across nodes → GPU-to-GPU without CPU.
**Routing Algorithms**
- **Deterministic**: Same path for same source-destination → simple, may cause congestion.
- **Adaptive**: Route based on current congestion → better utilization, harder to implement.
- **Minimal**: Shortest path only. **Non-minimal**: May take longer paths to avoid congestion.
Interconnect topology is **a defining architectural choice for any large-scale parallel system** — the topology determines the communication performance envelope within which all parallel algorithms must operate, making it one of the first and most consequential decisions in supercomputer and data center design.