parallel io file system

**Parallel I/O and Parallel File Systems** are the **storage technologies and programming interfaces that enable hundreds to thousands of compute nodes to read and write data simultaneously at aggregate bandwidths of hundreds of GB/s to TB/s — solving the I/O bottleneck that occurs when massively parallel computations must checkpoint state, read input datasets, or write results to persistent storage**. **The Parallel I/O Problem** A 10,000-node scientific simulation produces 100 TB of checkpoint data every 30 minutes. Writing 100 TB through a single file server at 10 GB/s takes 2.8 hours — longer than the compute interval. Parallel I/O distributes the data across hundreds of storage servers (Object Storage Targets in Lustre terminology), enabling aggregate bandwidth that scales with the number of servers. **Parallel File Systems** - **Lustre**: The dominant HPC parallel file system. Separates metadata (file names, permissions — stored on MDS) from file data (stored on OSTs, striped across multiple servers). A single file can be striped across 100+ OSTs, enabling 100+ GB/s bandwidth for a single file. Used on most TOP500 supercomputers. - **GPFS/Spectrum Scale (IBM)**: Block-based parallel file system with distributed locking. Provides POSIX semantics with strong consistency. Scales to tens of thousands of nodes. - **BeeGFS**: Open-source parallel file system designed for simplicity. Separates metadata and storage targets similar to Lustre. Popular in academic and mid-scale HPC clusters. **I/O Middleware** - **MPI-IO**: Part of the MPI-2 standard. Provides collective I/O operations where all processes in a communicator access a shared file with coordinated patterns. Collective I/O merges many small, non-contiguous accesses into fewer large sequential accesses — dramatically improving bandwidth. - **HDF5 (Hierarchical Data Format)**: Self-describing data format with a parallel I/O backend (phdf5) that uses MPI-IO. Supports multidimensional arrays, compression, chunking, and metadata — the standard for scientific data storage. - **NetCDF-4**: Built on HDF5, provides a simpler interface for array-oriented data. Standard for climate, weather, and oceanographic data. - **ADIOS2**: High-performance I/O framework with support for multiple backend engines (file, streaming, in-situ analysis). Provides producer-consumer data staging for workflow pipelines. **I/O Optimization Strategies** - **Collective I/O (Two-Phase)**: Processes exchange data among themselves to create large contiguous I/O requests, then a subset (aggregators) performs the actual file system I/O. Converts millions of small random accesses into hundreds of large sequential accesses. - **Subfiling**: Instead of one shared file, each process (or group) writes a separate file. Eliminates lock contention at the file system level. Data must be reassembled for post-processing. - **Burst Buffers**: Fast intermediate storage (SSD-based) absorbing bursty checkpoint writes at local speed, then draining to the parallel file system asynchronously. Decouples compute from I/O. Parallel I/O is **the storage infrastructure that prevents data movement from being the bottleneck of parallel computing** — because even the fastest computation is worthless if it takes longer to save the results than it took to compute them.