mpi collective operations
**MPI Collective Operations** are **communication patterns where all processes in a communicator participate simultaneously** — implementing broadcast, scatter, gather, reduce, and all-to-all operations essential for distributed memory parallel computing.
**Point-to-Point vs. Collective**
- Point-to-point: `MPI_Send` / `MPI_Recv` between two specific processes.
- Collective: All processes in communicator participate — synchronization implied.
- Collective operations are more efficient and easier to reason about than manual P2P.
**Core Collective Operations**
**MPI_Bcast (Broadcast)**:
```c
MPI_Bcast(buffer, count, MPI_INT, root, MPI_COMM_WORLD);
```
- Root sends buffer to all other processes.
- Used for: Broadcasting parameters, model weights.
**MPI_Scatter / MPI_Gather**:
- Scatter: Root sends different data to each process (work distribution).
- Gather: Each process sends data to root (result collection).
- MPI_Scatterv / Gatherv: Variable-length messages per process.
**MPI_Reduce**:
```c
MPI_Reduce(send, recv, count, MPI_DOUBLE, MPI_SUM, root, MPI_COMM_WORLD);
```
- Combine values from all processes using operation (SUM, MAX, MIN, PROD) → result at root.
**MPI_Allreduce**:
- Like Reduce but result available at ALL processes.
- Essential for distributed training: Sum gradients across all GPUs.
- Ring Allreduce: Most efficient algorithm — O(N) bandwidth, O(log N) latency.
**MPI_Alltoall**:
- Every process sends unique data to every other process.
- Used for: Matrix transpose, FFT butterfly, dense database joins.
- Most expensive collective: O(P²) messages in naive implementation.
**Algorithm Implementations**
- **Butterfly (Recursive Halving/Doubling)**: Optimal for small counts.
- **Ring**: Optimal bandwidth for large messages (allreduce, allgather).
- **Binomial Tree**: Optimal for broadcast/reduce in latency-dominated regime.
**Non-Blocking Collectives**
```c
MPI_Request req;
MPI_Iallreduce(sendbuf, recvbuf, count, dtype, op, comm, &req);
// Overlap computation here
MPI_Wait(&req, MPI_STATUS_IGNORE);
```
- Allows overlap of communication with computation — critical for scaling efficiency.
MPI collective operations are **the communication backbone of HPC and distributed training** — efficient collective implementations (MVAPICH, OpenMPI, NCCL) are what allow hundreds to thousands of GPUs to train LLMs together at near-linear efficiency.