Parallel I/O and Storage Systems are the hardware and software architectures that enable multiple processes to simultaneously read and write data to storage — essential for HPC and data-intensive computing where sequential I/O becomes the bottleneck, with parallel file systems like Lustre and GPFS distributing data across hundreds of storage servers to deliver aggregate bandwidths of terabytes per second.

The I/O Bottleneck

• Modern HPC: Thousands of cores computing at PFLOPS → need TB/s of data I/O.
• Single SSD: ~7 GB/s (NVMe Gen5). Single HDD: ~200 MB/s.
• 10,000 processes checkpoint at once → need 100+ GB/s aggregate.
• Without parallel I/O: Processes serialize at storage → compute idle → massive waste.

Parallel File Systems

Lustre Architecture

• MDS (Metadata Server): Handles namespace operations (open, stat, mkdir).
• OSS (Object Storage Server): Serves data to/from OSTs.
• OST (Object Storage Target): Physical storage (disk arrays) — typically dozens to hundreds.
• Striping: File split across multiple OSTs — each process reads from different OST simultaneously.
• Stripe size: Configurable (typically 1-4 MB).
• Stripe count: Number of OSTs per file (8-64 typical for large files).

MPI-IO (Standard Parallel I/O API)

• Extends MPI with file I/O operations: MPI_File_open, MPI_File_read_all, MPI_File_write_at.
• Collective I/O: All processes coordinate I/O → library optimizes access pattern.
• Two-phase I/O: Aggregate small scattered accesses into large sequential accesses → 10-100x speedup.
• Data sieving: Read a contiguous chunk, extract needed data → reduces number of I/O operations.

HDF5 Parallel

• High-level data format for scientific data (hierarchical, self-describing).
• Parallel HDF5: Built on MPI-IO — multiple processes write to same HDF5 file simultaneously.
• Chunking: Dataset divided into chunks — different processes write different chunks.
• Compression: Per-chunk compression — parallel decompression possible.

I/O Optimization Strategies

Parallel I/O and storage systems are the critical data backbone for computational science — without them, the raw computing power of modern supercomputers would be stranded waiting for data, making parallel storage architecture as important as the compute architecture for overall system performance.