io parallelism
**Parallel I/O and File Systems** encompasses the **techniques and systems for achieving high-bandwidth storage access by distributing file data (striping) across multiple storage servers and disks, enabling concurrent I/O from hundreds to thousands of compute nodes** — essential for scientific computing, AI training, and large-scale data analytics where I/O bottlenecks can dominate total application time.
**The I/O Bottleneck**: A single disk provides ~200 MB/s sequential bandwidth. A single NVMe SSD provides ~7 GB/s. But a large HPC application running on 1000 nodes may need 1+ TB/s aggregate bandwidth. Parallel file systems achieve this by striping data across thousands of storage targets.
**Parallel File System Architecture**:
| Component | Function | Example |
|-----------|----------|----------|
| **Metadata servers (MDS)** | Directory ops, file attributes | Lustre MDT, GPFS mmfsd |
| **Object storage servers (OSS)** | Store file data stripes | Lustre OST, BeeGFS storage |
| **Clients** | Parallel access from compute nodes | POSIX client, FUSE mount |
| **Network** | High-bandwidth interconnect | InfiniBand, 100GbE |
**File Striping**: A file is divided into fixed-size chunks (stripe units, typically 1-4 MB) distributed round-robin across multiple storage targets. A file with 1 MB stripe across 100 OSTs gets 100x the bandwidth of a single OST. Stripe count and size are tunable per file or directory — small files benefit from low stripe count (avoids metadata overhead), large files from high stripe count.
**Key Systems**:
- **Lustre**: Dominant HPC parallel file system. Separates metadata (MDS) from data (OSS). Supports file-level striping with progressive file layouts. Scalable to exabyte capacity with thousands of OSTs. POSIX-compliant.
- **GPFS/Spectrum Scale (IBM)**: Block-level distributed file system with shared-disk architecture. Every node can be both client and server. Strong consistency, high metadata performance. Common in enterprise HPC and AI.
- **BeeGFS**: Lightweight parallel file system with separate metadata and storage servers. Easy deployment. Gaining adoption in AI/ML clusters.
- **DAOS**: Intel's next-generation storage system targeting NVMe and persistent memory. Bypasses kernel and POSIX for lowest latency. Designed for exascale.
**MPI-IO**: The standard parallel I/O interface for HPC applications. Key concepts: **collective I/O** — processes coordinate I/O operations to merge many small requests into fewer large ones (two-phase I/O); **file views** — each process describes its access pattern using MPI datatypes, enabling non-contiguous I/O without performance loss; **hints** — tuning parameters (striping, buffering, aggregator count) communicated through MPI_Info objects.
**I/O Optimization Strategies**: **Aggregate small I/O** into large writes (collective I/O, buffered I/O); **align access** to stripe boundaries to avoid lock contention; **use dedicated I/O nodes** (burst buffers, I/O forwarding) to absorb bursty write patterns; **data staging** — stage checkpoint data to fast local NVMe before flushing to parallel FS; and **avoid metadata storms** — directory listing or stat() from thousands of nodes simultaneously overwhelms MDS.
**Parallel I/O is the often-overlooked third leg of HPC performance (alongside compute and communication) — a perfectly optimized computation that writes results through a serial I/O bottleneck wastes all the parallelism it worked so hard to achieve.**