Advanced MPI Communication encompasses sophisticated messaging primitives beyond basic send/receive, including persistent requests for reduced overhead, one-sided remote-memory-access patterns, and specialized datatype handling for irregular communication.

MPI Persistent Requests

• Persistent Send/Recv: Pre-allocate send/recv request (MPI_Send_init, MPI_Recv_init) with parameters (buffer, count, datatype, dest, tag). Reuse request in tight loops.
• Performance Benefit: Request initialization overhead amortized across multiple uses. Typical overhead reduction: 20-40% for bandwidth-limited messages.
• Usage Pattern: Start/complete cycle (MPI_Start, MPI_Wait). Multiple requests can be started (MPI_Startall) enabling pipelined communication.
• Compared to Non-Persistent: Each send/recv allocates request (small overhead but accumulates). Persistent requests ~5-10% faster in tight loops.

One-Sided Communication (Remote Memory Access, RMA)

• MPI Window Creation: MPI_Win_create(base, size, ...) registers memory region for RMA access. Other processes can read/write this window.
• RMA Operations: MPI_Put (write remote memory), MPI_Get (read remote memory), MPI_Accumulate (atomic operation on remote memory).
• Advantages: Sender initiates operation (PUT/GET) without target blocking. Sender knows when operation complete (local semantics). Enables asynchronous communication.
• Use Cases: Producer-consumer, work-stealing, load-balancing algorithms naturally express via RMA.

MPI Window Synchronization Semantics

• Fence Synchronization: MPI_Win_fence() acts as collective barrier (all processes in window). Ensures previous RMA operations completed globally.
• Post-Wait-Complete-Wait (PSCW): More flexible synchronization. MPI_Win_post(), MPI_Win_start(), MPI_Win_complete(), MPI_Win_wait(). Processes indicate participation, synchronize only when needed.
• Lock Synchronization: MPI_Win_lock() acquires exclusive/shared lock on target process. MPI_Win_unlock() releases. Enables fine-grained mutual exclusion.
• Memory Model: Fence: all processes agree on consistency. Lock: only target process sees consistent view. Pipelining: process-specific synchronization.

Derived Datatypes and Communication of Non-Contiguous Data

• Contiguous Datatype: MPI_FLOAT, MPI_INT, etc. communicate single array in memory.
• Vector Datatype: MPI_Type_vector(count, blocklen, stride, base_type) communicates evenly-spaced blocks. Example: column of matrix (stride = row_width).
• Indexed Datatype: MPI_Type_indexed(count, array_of_blocklengths, array_of_displacements) arbitrary displacements. Example: sparse matrix rows.
• Struct Datatype: MPI_Type_create_struct() combines multiple types with offsets. Example: structure containing integer + float fields.

Derived Datatype Usage

• MPI_Type_commit(): Finalize datatype definition before use. Commit enables compiler optimizations (e.g., compute contiguous regions).
• Packing Advantage: Derived datatype reduces host-CPU overhead vs manual packing/unpacking. Single MPI call vs loop of multiple calls.
• Subarray Extraction: MPI_Type_create_subarray() extracts rectangular region of N-dimensional array. Useful for domain decomposition (decompose 3D domain into 1D slices).

Neighborhood Collectives (MPI 3.0+)

• MPI_Neighbor_allgather: Local gather from neighbors (defined by topology/graph). Replaces global allgather for sparse communication patterns.
• MPI_Neighbor_alltoall: Local all-to-all (each rank sends to all neighbors, receives from all). Efficient for stencil computations.
• Topology Definition: MPI_Dist_graph_create() defines custom neighbor topology (sparse directed graph). Enables application-specific communication patterns.
• Optimization Opportunity: Neighborhood collectives permit more aggressive optimization (fewer ranks participate, topology-aware routing).

MPI-4 Features and Enhancements

• Persistent Collectives: MPI_Allreduce_init() similar to persistent send/recv. Pre-allocate collective request, reuse in loops.
• Partitioned Point-to-Point: Send/recv partitioned into smaller sub-messages, enabling overlap across multiple messages.
• Request-Based Collectives: Non-blocking collectives return request immediately. Enable pipelined collective operations across multiple pairs.
• Topology-Aware Mapping: Queries machine topology, maps ranks to optimize communication locality (reduce inter-socket/inter-switch traffic).

Real-World Optimization Strategies

• Double Buffering: Alternate between two buffers for ping-pong communication. While GPU computes buffer N, GPU transfers buffer N+1 to host asynchronously.
• Batching: Collect multiple small messages, send single large message. Reduces overhead (fewer syscalls, network headers).
• Stencil Optimization: Halos (boundary rows/cols) communicated separately from bulk. Computation on interior while edges exchange.