parallel io hpc lustre gpfs

**Parallel I/O and File Systems** are **the storage infrastructure technologies that enable thousands of compute nodes to simultaneously read and write data to shared storage at aggregate bandwidths exceeding terabytes per second — essential for HPC simulations, AI training data pipelines, and scientific data management where I/O performance directly impacts time-to-solution**. **Parallel File System Architecture:** - **Lustre**: open-source parallel file system dominating HPC — separates metadata (MDS/MDT) from data (OSS/OST) with data striped across multiple Object Storage Targets; single file can span hundreds of OSTs achieving >1 TB/s aggregate bandwidth - **GPFS/Spectrum Scale**: IBM's enterprise parallel file system — shared-disk architecture where all nodes directly access storage through SAN fabric; provides POSIX compliance with distributed locking for consistency - **BeeGFS**: high-performance parallel file system designed for simplicity — metadata and storage services can run on same or separate nodes; popular in university and AI training clusters - **CephFS**: distributed file system built on Ceph's RADOS object store — dynamic subtree partitioning for metadata scalability; increasingly used for AI and analytics workloads **MPI-IO and Collective I/O:** - **MPI_File_read/write**: individual I/O operations from each MPI rank — simple but can create millions of small I/O requests that overwhelm the file system metadata server - **Collective I/O (MPI_File_read_all)**: aggregates I/O requests from all ranks and performs a optimized number of large, contiguous reads/writes — two-phase I/O: aggregator ranks collect requests, perform bulk I/O, and redistribute data to requesting ranks - **File Views**: MPI_File_set_view defines each rank's portion of data using MPI datatypes — enables each rank to address its data using local indices while MPI translates to global file positions - **Striping Optimization**: align file stripe size and stripe count with I/O access patterns — larger stripes (4-16 MB) favor large sequential transfers; more OSTs increase aggregate bandwidth but add metadata overhead **I/O Optimization Techniques:** - **Burst Buffer**: NVMe-based intermediate storage tier between compute nodes and parallel file system — absorbs bursty checkpoint writes at local speeds (50+ GB/s per node), drains to parallel file system asynchronously; Cray DataWarp and DDN IME are examples - **Data Staging**: pre-load datasets from parallel file system to local NVMe or burst buffer before computation begins — eliminates I/O latency during compute phases; essential for AI training with repeated dataset passes - **I/O Aggregation**: reduce number of file system operations by buffering and combining small writes into large ones — application-level buffering or I/O middleware (ADIOS, HDF5) provides transparent aggregation - **Asynchronous I/O**: overlap I/O with computation using non-blocking writes and double buffering — while one buffer is being written to storage, computation fills the next buffer **Parallel I/O engineering is the often-overlooked performance discipline that determines whether exascale simulations spend their time computing or waiting for storage — a well-tuned I/O subsystem achieves 60-80% of peak storage bandwidth, while naive I/O patterns may achieve less than 1%.**