Advanced MPI Communication encompasses sophisticated messaging primitives beyond basic send/receive, including persistent requests for reduced overhead, one-sided remote-memory-access patterns, and specialized datatype handling for irregular communication.
MPI Persistent Requests
- Persistent Send/Recv: Pre-allocate send/recv request (MPI_Send_init, MPI_Recv_init) with parameters (buffer, count, datatype, dest, tag). Reuse request in tight loops.
- Performance Benefit: Request initialization overhead amortized across multiple uses. Typical overhead reduction: 20-40% for bandwidth-limited messages.
- Usage Pattern: Start/complete cycle (MPI_Start, MPI_Wait). Multiple requests can be started (MPI_Startall) enabling pipelined communication.
- Compared to Non-Persistent: Each send/recv allocates request (small overhead but accumulates). Persistent requests ~5-10% faster in tight loops.
One-Sided Communication (Remote Memory Access, RMA)
- MPI Window Creation: MPI_Win_create(base, size, ...) registers memory region for RMA access. Other processes can read/write this window.
- RMA Operations: MPI_Put (write remote memory), MPI_Get (read remote memory), MPI_Accumulate (atomic operation on remote memory).
- Advantages: Sender initiates operation (PUT/GET) without target blocking. Sender knows when operation complete (local semantics). Enables asynchronous communication.
- Use Cases: Producer-consumer, work-stealing, load-balancing algorithms naturally express via RMA.
MPI Window Synchronization Semantics
- Fence Synchronization: MPI_Win_fence() acts as collective barrier (all processes in window). Ensures previous RMA operations completed globally.
- Post-Wait-Complete-Wait (PSCW): More flexible synchronization. MPI_Win_post(), MPI_Win_start(), MPI_Win_complete(), MPI_Win_wait(). Processes indicate participation, synchronize only when needed.
- Lock Synchronization: MPI_Win_lock() acquires exclusive/shared lock on target process. MPI_Win_unlock() releases. Enables fine-grained mutual exclusion.
- Memory Model: Fence: all processes agree on consistency. Lock: only target process sees consistent view. Pipelining: process-specific synchronization.
Derived Datatypes and Communication of Non-Contiguous Data
- Contiguous Datatype: MPI_FLOAT, MPI_INT, etc. communicate single array in memory.
- Vector Datatype: MPI_Type_vector(count, blocklen, stride, base_type) communicates evenly-spaced blocks. Example: column of matrix (stride = row_width).
- Indexed Datatype: MPI_Type_indexed(count, array_of_blocklengths, array_of_displacements) arbitrary displacements. Example: sparse matrix rows.
- Struct Datatype: MPI_Type_create_struct() combines multiple types with offsets. Example: structure containing integer + float fields.
Derived Datatype Usage
- MPI_Type_commit(): Finalize datatype definition before use. Commit enables compiler optimizations (e.g., compute contiguous regions).
- Packing Advantage: Derived datatype reduces host-CPU overhead vs manual packing/unpacking. Single MPI call vs loop of multiple calls.
- Subarray Extraction: MPI_Type_create_subarray() extracts rectangular region of N-dimensional array. Useful for domain decomposition (decompose 3D domain into 1D slices).
Neighborhood Collectives (MPI 3.0+)
- MPI_Neighbor_allgather: Local gather from neighbors (defined by topology/graph). Replaces global allgather for sparse communication patterns.
- MPI_Neighbor_alltoall: Local all-to-all (each rank sends to all neighbors, receives from all). Efficient for stencil computations.
- Topology Definition: MPI_Dist_graph_create() defines custom neighbor topology (sparse directed graph). Enables application-specific communication patterns.
- Optimization Opportunity: Neighborhood collectives permit more aggressive optimization (fewer ranks participate, topology-aware routing).
MPI-4 Features and Enhancements
- Persistent Collectives: MPI_Allreduce_init() similar to persistent send/recv. Pre-allocate collective request, reuse in loops.
- Partitioned Point-to-Point: Send/recv partitioned into smaller sub-messages, enabling overlap across multiple messages.
- Request-Based Collectives: Non-blocking collectives return request immediately. Enable pipelined collective operations across multiple pairs.
- Topology-Aware Mapping: Queries machine topology, maps ranks to optimize communication locality (reduce inter-socket/inter-switch traffic).
Real-World Optimization Strategies
- Double Buffering: Alternate between two buffers for ping-pong communication. While GPU computes buffer N, GPU transfers buffer N+1 to host asynchronously.
- Batching: Collect multiple small messages, send single large message. Reduces overhead (fewer syscalls, network headers).
- Stencil Optimization: Halos (boundary rows/cols) communicated separately from bulk. Computation on interior while edges exchange.