mpi collective communication
**MPI Collective Communication Operations** are the **coordinated multi-process communication patterns where all (or a defined subset of) processes in a communicator participate simultaneously in data exchange — including broadcast, reduce, allreduce, scatter, gather, allgather, and alltoall — which are the dominant communication cost in most parallel scientific applications and whose algorithmic implementation determines whether communication scales efficiently to thousands of nodes**.
**Core Collective Operations**
| Operation | Description | Data Movement |
|-----------|-------------|---------------|
| **Broadcast** | One process sends to all | 1 → N |
| **Reduce** | All contribute, one receives result | N → 1 |
| **Allreduce** | Reduce + broadcast result to all | N → N |
| **Scatter** | One distributes unique parts to each | 1 → N (unique) |
| **Gather** | Each sends unique part to one | N → 1 (concatenate) |
| **Allgather** | Each sends its part, all receive full | N → N (concatenate) |
| **Alltoall** | Each sends unique data to every other | N → N (personalized) |
**Allreduce: The Most Critical Collective**
Allreduce (sum/max/min across all processes, result available to all) dominates distributed deep learning (gradient synchronization) and iterative solvers (global residual computation). Its implementation determines training throughput.
**Allreduce Algorithms**
- **Ring Allreduce**: Processes are arranged in a logical ring. Data is segmented into P chunks. Each process sends one chunk to its right neighbor and receives from its left, accumulating partial sums. After 2(P-1) steps, all processes have the complete result. Bandwidth cost: 2(P-1)/P × N bytes — approaches 2N regardless of P. Optimal bandwidth utilization but latency grows as O(P).
- **Recursive Halving-Doubling**: Processes pair up, exchange and reduce data at each step. After log2(P) steps, each process has a portion of the result. Then a reverse (doubling) phase distributes the result. Total cost: O(log P × α + N × log P × β) — better latency than ring for small messages.
- **Tree (Binomial) Reduce + Broadcast**: Reduce to root via binomial tree, then broadcast the result. Simple but root becomes a bottleneck for large messages.
- **NCCL (NVIDIA Collective Communications Library)**: Optimized for GPU clusters using NVLink/NVSwitch topology-aware algorithms. Uses ring or tree algorithms mapped to the physical NVLink rings, achieving near-peak NVLink bandwidth (900 GB/s on DGX H100).
**Overlap with Computation**
Non-blocking collectives (MPI_Iallreduce) allow computation to proceed while the collective executes in the background. This is essential for hiding communication latency: start the allreduce of layer N's gradients while computing layer N-1's backward pass.
MPI Collective Communication is **the coordination language of parallel computing** — every parallel algorithm that needs global agreement, global data redistribution, or global reduction depends on these primitives, and their efficient implementation is what separates a cluster that scales from one that saturates.