parallel io storage

**Parallel I/O and Storage Systems** are the **hardware and software architectures that enable multiple processes to simultaneously read and write data to storage** — essential for HPC and data-intensive computing where sequential I/O becomes the bottleneck, with parallel file systems like Lustre and GPFS distributing data across hundreds of storage servers to deliver aggregate bandwidths of terabytes per second. **The I/O Bottleneck** - Modern HPC: Thousands of cores computing at PFLOPS → need TB/s of data I/O. - Single SSD: ~7 GB/s (NVMe Gen5). Single HDD: ~200 MB/s. - 10,000 processes checkpoint at once → need 100+ GB/s aggregate. - Without parallel I/O: Processes serialize at storage → compute idle → massive waste. **Parallel File Systems** | System | Developer | Stripe Unit | Max Bandwidth | Typical Use | |--------|----------|------------|-------------|------------| | Lustre | OpenSFS | Object Storage Targets (OSTs) | 1+ TB/s | HPC, national labs | | GPFS/Spectrum Scale | IBM | Network Shared Disk | 1+ TB/s | Enterprise HPC | | BeeGFS | ThinkParQ | Storage targets | 100+ GB/s | Research clusters | | CephFS | Red Hat | RADOS objects | 100+ GB/s | Cloud, HPC | | DAOS | Intel | SCM + NVMe | High IOPS + bandwidth | Exascale HPC | **Lustre Architecture** - **MDS (Metadata Server)**: Handles namespace operations (open, stat, mkdir). - **OSS (Object Storage Server)**: Serves data to/from OSTs. - **OST (Object Storage Target)**: Physical storage (disk arrays) — typically dozens to hundreds. - **Striping**: File split across multiple OSTs — each process reads from different OST simultaneously. - Stripe size: Configurable (typically 1-4 MB). - Stripe count: Number of OSTs per file (8-64 typical for large files). **MPI-IO (Standard Parallel I/O API)** - Extends MPI with file I/O operations: `MPI_File_open`, `MPI_File_read_all`, `MPI_File_write_at`. - **Collective I/O**: All processes coordinate I/O → library optimizes access pattern. - **Two-phase I/O**: Aggregate small scattered accesses into large sequential accesses → 10-100x speedup. - **Data sieving**: Read a contiguous chunk, extract needed data → reduces number of I/O operations. **HDF5 Parallel** - High-level data format for scientific data (hierarchical, self-describing). - Parallel HDF5: Built on MPI-IO — multiple processes write to same HDF5 file simultaneously. - Chunking: Dataset divided into chunks — different processes write different chunks. - Compression: Per-chunk compression — parallel decompression possible. **I/O Optimization Strategies** | Strategy | How | Benefit | |----------|-----|--------| | Increase stripe count | File spread across more OSTs | Higher aggregate bandwidth | | Align to stripe boundary | Process I/O aligned to stripe units | Fewer cross-OST operations | | Collective I/O | Coordinate processes → merge small I/Os | Reduce metadata + seek overhead | | Burst buffer | Fast tier (NVMe) absorbs bursts | Smooth out I/O peaks | | Asynchronous I/O | Overlap I/O with computation | Hide I/O latency | Parallel I/O and storage systems are **the critical data backbone for computational science** — without them, the raw computing power of modern supercomputers would be stranded waiting for data, making parallel storage architecture as important as the compute architecture for overall system performance.