mpi basics

**MPI (Message Passing Interface)** — the standard programming model for distributed-memory parallel computing, where each process has its own memory and communicates by sending messages. **Core Concepts** - Each MPI process has a unique **rank** (0 to N-1) - Processes run on different cores or different machines - No shared memory — all data exchange through explicit messages - Communicator: Group of processes that can communicate (default: MPI_COMM_WORLD) **Essential Functions** - `MPI_Send(data, dest_rank)` — send data to another process - `MPI_Recv(data, src_rank)` — receive data from another process - `MPI_Bcast` — one-to-all broadcast - `MPI_Reduce` — combine data from all processes (sum, max, etc.) - `MPI_Scatter` / `MPI_Gather` — distribute/collect data portions - `MPI_Allreduce` — reduce + broadcast result to all (most used collective) **Usage** ``` mpirun -np 128 ./my_simulation ``` Runs 128 processes across available nodes. **Where MPI Is Used** - Scientific simulation (weather, molecular dynamics, CFD) - HPC clusters (Top500 supercomputers) - Distributed deep learning training (combined with NCCL for GPU communication) **MPI** remains the backbone of large-scale parallel computing after 30+ years — virtually all HPC applications use it.