mpi derived datatypes

**MPI Derived Datatypes** are **user-defined type descriptors that enable efficient communication of noncontiguous, heterogeneous, or structured data without manual packing into contiguous buffers — allowing MPI to directly access scattered memory locations during send/receive operations with optimal zero-copy performance on supported networks**. **Type Constructor Hierarchy:** - **MPI_Type_contiguous**: creates a type from N consecutive copies of an existing type — simplest constructor, equivalent to a C array - **MPI_Type_vector/hvector**: describes N blocks of count elements with fixed stride between blocks — ideal for matrix columns, subarray slices, and strided grid data; hvector specifies stride in bytes for heterogeneous layouts - **MPI_Type_indexed/hindexed**: each block has individually specified offset and size — handles irregular access patterns like sparse matrix rows or adaptive mesh element lists - **MPI_Type_create_struct**: most general constructor combining different base types at arbitrary byte offsets — maps directly to C structs with mixed types and padding **Zero-Copy Protocol:** - **Packing Avoidance**: when hardware supports scatter-gather (InfiniBand, Omni-Path), derived datatypes enable direct RDMA from noncontiguous memory without copying to intermediate buffers — eliminating the serialization overhead of MPI_Pack/MPI_Unpack - **Type Commit Optimization**: MPI_Type_commit analyzes the type map and selects the optimal data access strategy — pipelining scattered reads with network transfers for large messages - **Dataloop Representation**: internal representation of committed types as iteration patterns (loops over blocks with stride/offset) enables efficient traversal without per-element function calls - **Network Offload**: modern interconnects (UCX, libfabric) can offload derived datatype processing to the NIC for hardware-accelerated scatter-gather DMA **Common Patterns:** - **Matrix Subarray**: MPI_Type_create_subarray extracts an N-dimensional subblock from a larger array — used for halo exchange in structured grid codes, distributing 2D/3D domain decompositions - **Struct Serialization**: defining MPI types matching C/Fortran structs enables direct communication of record-oriented data without manual field-by-field packing - **Indexed Scatter**: MPI_Type_indexed with per-element offsets enables gather/scatter patterns — extracting boundary nodes from unstructured mesh data or communicating sparse vector entries **Performance Considerations:** - **Small Message Overhead**: for very small messages (<1 KB), the overhead of type traversal may exceed manual packing cost — benchmark before adopting derived types for latency-sensitive small messages - **Nested Type Depth**: deeply nested type constructors (types built from types built from types) can cause performance degradation in some MPI implementations — flattening to indexed types may help - **Memory Registration**: RDMA-based transports require memory registration for zero-copy; scattered pages may require multiple registrations, partially negating the benefit of avoiding packing MPI derived datatypes are **an essential abstraction for scientific computing that eliminates error-prone manual data serialization while enabling MPI implementations to optimize noncontiguous data transfer — achieving both programmer productivity and communication performance for complex distributed data structures**.