InfiniBand | ChipFoundryServices

Home› Knowledge Base› InfiniBand

InfiniBand is a high-bandwidth, low-latency networking technology using RDMA for GPU cluster communication — providing 200-400 Gbps per port with microsecond latencies, InfiniBand is the interconnect of choice for large-scale AI training where multi-node communication efficiency determines scaling effectiveness.

What Is InfiniBand?

Definition: High-performance networking fabric for clusters.
Technology: RDMA (Remote Direct Memory Access).
Vendor: NVIDIA/Mellanox (dominant).
Use Case: HPC, AI training, storage networks.

Why InfiniBand for AI

Bandwidth: 400 Gbps (NDR) vs. 100 Gbps Ethernet.
Latency: ~1 μs vs. ~10-50 μs Ethernet.
RDMA: Bypass CPU for GPU-to-GPU transfers.
Scaling: Efficient all-reduce across thousands of GPUs.
Proven: Used in largest AI training runs.

InfiniBand Generations

Speed Evolution:

Generation | Speed (per port) | Year
-----------|------------------|------
EDR        | 100 Gbps         | 2014
HDR        | 200 Gbps         | 2019
NDR        | 400 Gbps         | 2022
XDR        | 800 Gbps         | 2024
GDR        | 1600 Gbps        | Future

Comparison with Ethernet:

Aspect        | InfiniBand NDR | 400G Ethernet
--------------|----------------|---------------
Bandwidth     | 400 Gbps       | 400 Gbps
Latency       | ~1 μs          | ~10-50 μs
RDMA          | Native         | RoCE (extra)
Congestion    | Credit-based   | Drop-based
CPU overhead  | Minimal        | Higher
AI training   | Optimized      | Improving
Cost          | Higher         | Lower

RDMA Explained

How RDMA Works:

Traditional Network:
CPU → Copy to buffer → NIC → Network → NIC → Copy to buffer → CPU

RDMA:
GPU Memory → NIC → Network → NIC → GPU Memory
(CPU not involved, zero-copy)

GPU Direct RDMA:

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">InfiniBand — The AI Cluster Network</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">RDMA networking: bypass the kernel, write directly to remote GPU memory — 400-800 Gbps per port</text>

  <!-- === TOP: Network topology (fat tree) === -->
  <rect x="30" y="65" width="700" height="150" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="83" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Fat-Tree Topology (AI training cluster)</text>

  <!-- Spine switches (top tier) -->
  <rect x="240" y="92" width="55" height="22" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="267" y="107" fill="#fbbf24" font-size="7.5" text-anchor="middle">Spine</text>
  <rect x="320" y="92" width="55" height="22" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="347" y="107" fill="#fbbf24" font-size="7.5" text-anchor="middle">Spine</text>
  <rect x="400" y="92" width="55" height="22" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="427" y="107" fill="#fbbf24" font-size="7.5" text-anchor="middle">Spine</text>
  <rect x="480" y="92" width="55" height="22" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="507" y="107" fill="#fbbf24" font-size="7.5" text-anchor="middle">Spine</text>

  <!-- Leaf switches (middle tier) -->
  <rect x="100" y="128" width="55" height="22" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="1"/>
  <text x="127" y="143" fill="#c4b5fd" font-size="7.5" text-anchor="middle">Leaf</text>
  <rect x="200" y="128" width="55" height="22" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="1"/>
  <text x="227" y="143" fill="#c4b5fd" font-size="7.5" text-anchor="middle">Leaf</text>
  <rect x="340" y="128" width="55" height="22" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="1"/>
  <text x="367" y="143" fill="#c4b5fd" font-size="7.5" text-anchor="middle">Leaf</text>
  <rect x="460" y="128" width="55" height="22" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="1"/>
  <text x="487" y="143" fill="#c4b5fd" font-size="7.5" text-anchor="middle">Leaf</text>
  <rect x="580" y="128" width="55" height="22" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="1"/>
  <text x="607" y="143" fill="#c4b5fd" font-size="7.5" text-anchor="middle">Leaf</text>

  <!-- Connections spine↔leaf -->
  <g stroke="#6b7684" stroke-width="0.5" opacity="0.4">
    <line x1="127" y1="128" x2="267" y2="114"/><line x1="127" y1="128" x2="347" y2="114"/>
    <line x1="227" y1="128" x2="267" y2="114"/><line x1="227" y1="128" x2="427" y2="114"/>
    <line x1="367" y1="128" x2="347" y2="114"/><line x1="367" y1="128" x2="507" y2="114"/>
    <line x1="487" y1="128" x2="427" y2="114"/><line x1="487" y1="128" x2="507" y2="114"/>
    <line x1="607" y1="128" x2="427" y2="114"/><line x1="607" y1="128" x2="507" y2="114"/>
  </g>

  <!-- GPU nodes (bottom) -->
  <g fill="#122040" stroke="#60a5fa" stroke-width="0.8">
    <rect x="55" y="162" width="40" height="20" rx="2"/><rect x="100" y="162" width="40" height="20" rx="2"/>
    <rect x="155" y="162" width="40" height="20" rx="2"/><rect x="200" y="162" width="40" height="20" rx="2"/>
    <rect x="290" y="162" width="40" height="20" rx="2"/><rect x="335" y="162" width="40" height="20" rx="2"/>
    <rect x="410" y="162" width="40" height="20" rx="2"/><rect x="455" y="162" width="40" height="20" rx="2"/>
    <rect x="535" y="162" width="40" height="20" rx="2"/><rect x="580" y="162" width="40" height="20" rx="2"/>
    <rect x="625" y="162" width="40" height="20" rx="2"/><rect x="670" y="162" width="40" height="20" rx="2"/>
  </g>
  <text x="127" y="196" fill="#93c5fd" font-size="7" text-anchor="middle">DGX nodes (8 GPU each)</text>
  <text x="450" y="196" fill="#93c5fd" font-size="7" text-anchor="middle">DGX nodes</text>

  <!-- Connections leaf↔nodes -->
  <g stroke="#3b82f6" stroke-width="0.5" opacity="0.4">
    <line x1="75" y1="162" x2="127" y2="150"/><line x1="120" y1="162" x2="127" y2="150"/>
    <line x1="175" y1="162" x2="227" y2="150"/><line x1="220" y1="162" x2="227" y2="150"/>
    <line x1="310" y1="162" x2="367" y2="150"/><line x1="355" y1="162" x2="367" y2="150"/>
    <line x1="430" y1="162" x2="487" y2="150"/><line x1="475" y1="162" x2="487" y2="150"/>
    <line x1="555" y1="162" x2="607" y2="150"/><line x1="600" y1="162" x2="607" y2="150"/>
  </g>

  <text x="380" y="210" fill="#fbbf24" font-size="8.5" text-anchor="middle">non-blocking fat-tree: every node can talk to every other at full BW simultaneously</text>

  <!-- === BOTTOM LEFT: RDMA concept === -->
  <rect x="30" y="225" width="340" height="120" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="243" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">RDMA — Zero-Copy, Kernel-Bypass</text>

  <!-- Traditional path (crossed out) -->
  <text x="50" y="263" fill="#f87171" font-size="8.5" text-decoration="line-through">TCP/IP: app → kernel → copy → NIC → wire → NIC → copy → kernel → app</text>

  <!-- RDMA path -->
  <text x="50" y="283" fill="#34d399" font-size="9">RDMA: app → NIC → wire → NIC → app (direct)</text>
  <text x="50" y="300" fill="#8b98a5" font-size="8.5">no kernel involvement, no data copies</text>
  <text x="50" y="316" fill="#8b98a5" font-size="8.5">latency: 1-2 µs (vs 50+ µs TCP)</text>
  <text x="50" y="332" fill="#6b7684" font-size="8">GPU-Direct RDMA: NIC writes directly to GPU HBM</text>
  <text x="50" y="346" fill="#6b7684" font-size="8">(skips CPU entirely for AllReduce)</text>

  <!-- === BOTTOM RIGHT: Speeds + generations === -->
  <rect x="385" y="225" width="345" height="120" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="557" y="243" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">InfiniBand Generations</text>

  <text x="405" y="263" fill="#6b7684" font-size="9">QDR (2008):   40 Gbps</text>
  <text x="405" y="279" fill="#6b7684" font-size="9">FDR (2011):   56 Gbps</text>
  <text x="405" y="295" fill="#8b98a5" font-size="9">EDR (2014):   100 Gbps</text>
  <text x="405" y="311" fill="#8b98a5" font-size="9">HDR (2018):   200 Gbps</text>
  <text x="405" y="327" fill="#60a5fa" font-size="9" font-weight="600">NDR (2022):   400 Gbps</text>
  <text x="405" y="343" fill="#fbbf24" font-size="9" font-weight="600">XDR (2025):   800 Gbps</text>

  <text x="600" y="279" fill="#6b7684" font-size="8">NVIDIA/Mellanox</text>
  <text x="600" y="295" fill="#6b7684" font-size="8">ConnectX HCA</text>
  <text x="600" y="311" fill="#6b7684" font-size="8">Quantum switches</text>
  <text x="600" y="327" fill="#8b98a5" font-size="8">current production</text>
  <text x="600" y="343" fill="#fbbf24" font-size="8">next gen</text>

  <!-- === BOTTOM: Why IB for AI === -->
  <rect x="30" y="355" width="700" height="60" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="373" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Why InfiniBand Dominates AI Training</text>
  <text x="50" y="393" fill="#8b98a5" font-size="9">AllReduce (gradient sync): 100K GPUs each send ~1GB/step → network must deliver TB/s aggregate</text>
  <text x="50" y="409" fill="#8b98a5" font-size="9">GPU-Direct RDMA + SHARP (in-network reduction): hardware-accelerated collective ops inside the switch</text>

  <!-- Key numbers -->
  <text x="380" y="432" fill="#e6edf3" font-size="9.5" text-anchor="middle" font-weight="600">Meta 24K GPU cluster: 3 tiers of NDR IB switches | xAI Colossus: 100K H100 on IB fabric</text>
  <text x="380" y="448" fill="#8b98a5" font-size="9" text-anchor="middle">Alternative: RoCE v2 (RDMA over Ethernet) — cheaper but worse tail latency. Broadcom/Arista compete here.</text>

  <text x="380" y="464" fill="#6b7684" font-size="11" text-anchor="middle">InfiniBand is the backplane of AI: 400K+ connections at microsecond latency — the highway between every GPU.</text>
</svg>

AI Training Infrastructure

Typical Large Cluster:

<svg viewBox="0 0 536 340" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto" role="img"><rect x="0" y="0" width="536" height="340" rx="12" fill="#0d1117"/><g font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,&quot;Liberation Mono&quot;,monospace" font-size="14"><text xml:space="preserve" x="20" y="31.7"><tspan fill="#6e7681">┌─────────────────────────────────────────────────────────┐</tspan></text><text xml:space="preserve" x="20" y="50.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">                    Spine Switches                       </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="69.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">  (InfiniBand NDR, high-radix, non-blocking)            </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="88.7"><tspan fill="#6e7681">└─────────────────────────────────────────────────────────┘</tspan></text><text xml:space="preserve" x="20" y="107.7"><tspan fill="#c9d1d9">           </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">         </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">         </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">         </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="126.7"><tspan fill="#c9d1d9">           </tspan><tspan fill="#6e7681">▼</tspan><tspan fill="#c9d1d9">         </tspan><tspan fill="#6e7681">▼</tspan><tspan fill="#c9d1d9">         </tspan><tspan fill="#6e7681">▼</tspan><tspan fill="#c9d1d9">         </tspan><tspan fill="#6e7681">▼</tspan></text><text xml:space="preserve" x="20" y="145.7"><tspan fill="#c9d1d9">     </tspan><tspan fill="#6e7681">┌─────────┐┌─────────┐┌─────────┐┌─────────┐</tspan></text><text xml:space="preserve" x="20" y="164.7"><tspan fill="#c9d1d9">     </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">  Leaf   </tspan><tspan fill="#6e7681">││</tspan><tspan fill="#c9d1d9">  Leaf   </tspan><tspan fill="#6e7681">││</tspan><tspan fill="#c9d1d9">  Leaf   </tspan><tspan fill="#6e7681">││</tspan><tspan fill="#c9d1d9">  Leaf   </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="183.7"><tspan fill="#c9d1d9">     </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> Switch  </tspan><tspan fill="#6e7681">││</tspan><tspan fill="#c9d1d9"> Switch  </tspan><tspan fill="#6e7681">││</tspan><tspan fill="#c9d1d9"> Switch  </tspan><tspan fill="#6e7681">││</tspan><tspan fill="#c9d1d9"> Switch  </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="202.7"><tspan fill="#c9d1d9">     </tspan><tspan fill="#6e7681">└────┬────┘└────┬────┘└────┬────┘└────┬────┘</tspan></text><text xml:space="preserve" x="20" y="221.7"><tspan fill="#c9d1d9">          </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">          </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">          </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">          </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="240.7"><tspan fill="#c9d1d9">    </tspan><tspan fill="#6e7681">┌─────┼─────┐</tspan><tspan fill="#c9d1d9">   ...        ...        ...</tspan></text><text xml:space="preserve" x="20" y="259.7"><tspan fill="#c9d1d9">    </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">     </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">     </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="278.7"><tspan fill="#6e7681">┌──────┐┌──────┐┌──────┐</tspan></text><text xml:space="preserve" x="20" y="297.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">DGX 1 </tspan><tspan fill="#6e7681">││</tspan><tspan fill="#c9d1d9">DGX 2 </tspan><tspan fill="#6e7681">││</tspan><tspan fill="#c9d1d9">DGX 3 </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">  (8 H100s each)</tspan></text><text xml:space="preserve" x="20" y="316.7"><tspan fill="#6e7681">└──────┘└──────┘└──────┘</tspan></text></g></svg>

NCCL with InfiniBand

import torch
import torch.distributed as dist
import os

# Set NCCL environment for InfiniBand
os.environ["NCCL_IB_DISABLE"] = "0"  # Enable InfiniBand
os.environ["NCCL_NET_GDR_LEVEL"] = "5"  # Enable GPUDirect

# Initialize distributed
dist.init_process_group(
    backend="nccl",
    init_method="env://",
)

# Training code - NCCL uses InfiniBand automatically
model = DistributedDataParallel(model)

Checking InfiniBand

# List InfiniBand devices
ibstat

# Show port status
ibstatus

# Check link speed
ibstat mlx5_0 | grep Rate

# Performance test
ib_write_bw -d mlx5_0

InfiniBand vs. Alternatives

Use Case              | Best Choice
----------------------|------------------
AI training (1000+ GPU) | InfiniBand NDR
Small clusters (<64 GPU)| Either (cost-dependent)
Cloud/flexibility     | Ethernet (easier)
Maximum performance   | InfiniBand
Budget constrained    | 400G Ethernet + RoCE

Cost Considerations

Component          | InfiniBand | 400G Ethernet
-------------------|------------|---------------
NIC/HCA            | $3-5K      | $1-2K
Switch (port)      | $500-1K    | $200-400
Total system cost  | Higher     | Lower
Performance/$      | Better at scale | Better for small

InfiniBand is the performance backbone of large-scale AI training — when training frontier models across thousands of GPUs, the efficiency of collective operations enabled by InfiniBand's low latency and RDMA capabilities directly determines how well training scales.

infinibandrdmanetworkhpcmellanoxclusterlatency

Explore 500+ Semiconductor & AI Topics

From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.

🔍 Search Topics 💬 Ask CFSGPT 📚 Browse All