distributed file system
**Distributed and Parallel File Systems (Lustre, HDFS, GPFS)** are the **storage systems that stripe files across many servers to provide aggregate I/O bandwidth and capacity far beyond any single storage node** — essential for HPC simulations that read/write terabytes of checkpoint data, ML training pipelines that stream petabytes of training data, and analytics workloads that process massive datasets, where parallel I/O at 100+ GB/s is a fundamental requirement.
**Why Distributed File Systems**
- Single NFS server: ~1-5 GB/s, ~100 TB capacity. Insufficient for HPC/ML workloads.
- Distributed FS: Aggregate bandwidth scales with number of servers.
- 100 storage servers × 5 GB/s each = 500 GB/s aggregate → reads entire dataset in seconds.
**Major Systems Comparison**
| System | Developer | Primary Use | Max Performance |
|--------|----------|------------|----------------|
| Lustre | OpenSFS | HPC, supercomputing | 1+ TB/s |
| GPFS/Spectrum Scale | IBM | Enterprise HPC | 500+ GB/s |
| HDFS | Apache | Big data (Hadoop/Spark) | 100+ GB/s |
| BeeGFS | ThinkParQ | ML training, HPC | 200+ GB/s |
| CephFS | Red Hat | Cloud, general purpose | 100+ GB/s |
| WekaFS | WEKA | ML training (NVMe-native) | 300+ GB/s |
**Lustre Architecture**
```svg
```
- **MDS (Metadata Server)**: Handles file names, directories, permissions.
- **OSS (Object Storage Server)**: Serves file data chunks.
- **OST (Object Storage Target)**: Physical storage volumes.
- **Striping**: Large files split across multiple OSTs → parallel I/O.
**HDFS Architecture**
- **NameNode**: Single metadata server (directories, block locations).
- **DataNodes**: Store 128MB blocks with 3× replication.
- Designed for: Large sequential reads/writes (MapReduce).
- Weakness: Small files, random access, low latency (not designed for these).
**I/O Patterns and File System Choice**
| Workload | I/O Pattern | Best File System |
|----------|------------|------------------|
| HPC simulation (CFD, molecular dynamics) | Large checkpoint writes, parallel reads | Lustre, GPFS |
| ML training (ImageNet, web data) | Random small reads, sequential large reads | WekaFS, Lustre, GPFS |
| Big data analytics (Spark) | Sequential scan, shuffle | HDFS, CephFS |
| AI model checkpointing | Periodic large writes (10-100 GB) | Lustre, GPFS |
| Genomics pipeline | Many small files + large BAM files | GPFS, BeeGFS |
**Performance Tuning**
| Technique | What | Impact |
|-----------|------|--------|
| Stripe count | Number of OSTs per file | More stripes → higher bandwidth |
| Stripe size | Bytes per OST before next | Match I/O request size |
| Client caching | Read-ahead and write-behind | Reduce metadata operations |
| Parallel I/O (MPI-IO) | Coordinated multi-process writes | Avoid lock contention |
Distributed file systems are **the storage backbone of every HPC center and AI training cluster** — without parallel file systems that can deliver hundreds of GB/s of aggregate bandwidth across thousands of concurrent readers, modern AI training runs would be bottlenecked by data loading rather than GPU computation, and HPC simulations would spend more time on I/O than on science.