parallel io file system

**Parallel I/O and File Systems** are the **storage infrastructure that enables thousands of compute nodes to simultaneously read and write data at aggregate bandwidths of hundreds of GB/s to TB/s — using parallel file systems (Lustre, GPFS/Spectrum Scale, BeeGFS) that stripe data across hundreds of storage servers and expose POSIX or MPI-IO interfaces, because HPC and AI workloads that generate petabytes of data per simulation run would bottleneck on serial I/O by orders of magnitude**. **Why Parallel I/O** A single storage server provides 1-5 GB/s sequential bandwidth. A supercomputer with 10,000 nodes running a climate simulation writes a 100 TB checkpoint every hour — requiring 28+ GB/s sustained. Only parallel I/O across many storage servers can achieve this. **Parallel File System Architecture** - **Lustre (Open-Source, most widely deployed in HPC)**: - **MDS (Metadata Server)**: Handles file creation, directory operations, permissions. One active MDS per filesystem (HA pair). Bottleneck for metadata-heavy workloads (many small files). - **OSS (Object Storage Server)**: Manages one or more OSTs (Object Storage Targets — typically RAID groups of disks or SSDs). Data is striped across OSTs for bandwidth. - **Client**: Mounts the filesystem on compute nodes. Reads/writes go directly to OSTs (no MDS in the data path). Striping: a single file is divided into stripes distributed round-robin across OSTs. - **Performance**: Top installations achieve 1+ TB/s aggregate bandwidth with 100+ OSTs. - **GPFS/Spectrum Scale (IBM)**: - Shared-disk model — all nodes access all disks through a SAN. - Distributed locking (token-based) for concurrent access. - Integrated tiering: hot data on SSDs, warm on HDDs, cold archived to tape. - Used in: enterprise HPC, AI training clusters, financial services. **MPI-IO** The parallel I/O interface for MPI programs: - **Collective I/O**: All processes in a communicator participate in a single I/O operation. The MPI library aggregates small, scattered requests into large, sequential I/O operations — dramatically improving effective bandwidth. - **Two-Phase I/O**: Phase 1: shuffle data among processes so that each process handles a contiguous range of the file. Phase 2: each process reads/writes its contiguous range. Converts random I/O to sequential. - **File Views**: Each process defines its view (subarray, distribution) of a shared file using MPI datatypes. MPI-IO uses views to compute optimal aggregation. **I/O Optimization Techniques** - **Striping Configuration**: Match stripe count and stripe size to workload. Large sequential writes benefit from high stripe count (all OSTs active). Small random I/O benefits from lower stripe count (reduce metadata overhead). - **Burst Buffer**: NVMe tier between compute nodes and parallel file system. Absorbs checkpoint writes at 100+ GB/s, drains to disk in the background. NERSC Perlmutter uses 30 PB Lustre + 1.5 PB NVMe burst buffer. - **I/O Forwarding**: Reduce the number of nodes accessing the file system simultaneously. I/O forwarding nodes aggregate requests from compute nodes, presenting fewer concurrent clients to the file system. Parallel I/O and File Systems are **the data infrastructure backbone of HPC and AI** — providing the bandwidth to feed data-hungry simulations and training runs, and the capacity to store the petabytes of results that drive scientific discovery and model development.