parallel file io

**Parallel File I/O** — reading and writing data across multiple storage devices and processes simultaneously, essential for HPC and large-scale data processing where sequential I/O is a bottleneck. **Why Parallel I/O?** - Single disk: ~200 MB/s sequential read - 100 disks in parallel: ~20 GB/s → 100x faster - Large-scale simulations and AI training generate/consume TB–PB of data **Parallel Filesystems** - **Lustre**: Most common HPC filesystem. Separates metadata (MDS) from data (OSS). Scales to 1000s of clients, PB+ storage, 1+ TB/s aggregate bandwidth - **GPFS/Spectrum Scale (IBM)**: Enterprise parallel filesystem. Strong metadata performance - **BeeGFS**: Open-source, easy to deploy. Popular for AI clusters - **WekaIO**: Flash-native parallel filesystem. Ultra-low latency **Striping** - Files split into chunks distributed across storage servers - Client reads/writes to multiple servers in parallel - Stripe size: 1-4 MB typical. Tunable for workload **Parallel I/O Libraries** - **MPI-IO**: Part of MPI standard. Collective I/O for coordinated access - **HDF5**: Self-describing scientific data format. Parallel HDF5 for multi-process access - **NetCDF**: Climate/weather data. Parallel variant available - **POSIX I/O**: Not parallel-aware → contention at filesystem level **Best Practices** - Large sequential writes >> many small random writes - Use collective I/O (aggregate small requests into large ones) - Match stripe count to number of writing processes **Parallel I/O** is often the overlooked bottleneck — a perfectly parallelized computation means nothing if data loading/saving can't keep up.