Parallel I/O and Parallel File Systems are the storage technologies and programming interfaces that enable hundreds to thousands of compute nodes to read and write data simultaneously at aggregate bandwidths of hundreds of GB/s to TB/s — solving the I/O bottleneck that occurs when massively parallel computations must checkpoint state, read input datasets, or write results to persistent storage.

The Parallel I/O Problem

A 10,000-node scientific simulation produces 100 TB of checkpoint data every 30 minutes. Writing 100 TB through a single file server at 10 GB/s takes 2.8 hours — longer than the compute interval. Parallel I/O distributes the data across hundreds of storage servers (Object Storage Targets in Lustre terminology), enabling aggregate bandwidth that scales with the number of servers.

Parallel File Systems

• Lustre: The dominant HPC parallel file system. Separates metadata (file names, permissions — stored on MDS) from file data (stored on OSTs, striped across multiple servers). A single file can be striped across 100+ OSTs, enabling 100+ GB/s bandwidth for a single file. Used on most TOP500 supercomputers.
• GPFS/Spectrum Scale (IBM): Block-based parallel file system with distributed locking. Provides POSIX semantics with strong consistency. Scales to tens of thousands of nodes.
• BeeGFS: Open-source parallel file system designed for simplicity. Separates metadata and storage targets similar to Lustre. Popular in academic and mid-scale HPC clusters.

I/O Middleware

• MPI-IO: Part of the MPI-2 standard. Provides collective I/O operations where all processes in a communicator access a shared file with coordinated patterns. Collective I/O merges many small, non-contiguous accesses into fewer large sequential accesses — dramatically improving bandwidth.
• HDF5 (Hierarchical Data Format): Self-describing data format with a parallel I/O backend (phdf5) that uses MPI-IO. Supports multidimensional arrays, compression, chunking, and metadata — the standard for scientific data storage.
• NetCDF-4: Built on HDF5, provides a simpler interface for array-oriented data. Standard for climate, weather, and oceanographic data.
• ADIOS2: High-performance I/O framework with support for multiple backend engines (file, streaming, in-situ analysis). Provides producer-consumer data staging for workflow pipelines.

I/O Optimization Strategies

• Collective I/O (Two-Phase): Processes exchange data among themselves to create large contiguous I/O requests, then a subset (aggregators) performs the actual file system I/O. Converts millions of small random accesses into hundreds of large sequential accesses.
• Subfiling: Instead of one shared file, each process (or group) writes a separate file. Eliminates lock contention at the file system level. Data must be reassembled for post-processing.
• Burst Buffers: Fast intermediate storage (SSD-based) absorbing bursty checkpoint writes at local speed, then draining to the parallel file system asynchronously. Decouples compute from I/O.

Parallel I/O is the storage infrastructure that prevents data movement from being the bottleneck of parallel computing — because even the fastest computation is worthless if it takes longer to save the results than it took to compute them.