checkpoint sharding

**Checkpoint sharding** is the **distributed save approach where checkpoint state is partitioned across multiple files or nodes** - it avoids single-file bottlenecks and enables parallel checkpoint I/O for very large model states. **What Is Checkpoint sharding?** - **Definition**: Splitting checkpoint data into shards aligned to data-parallel ranks or model partitions. - **Scale Context**: Essential when full model state is too large for efficient single-stream writes. - **Read Path**: Restore requires coordinated loading and reassembly of all shard components. - **Metadata Layer**: A manifest maps shard locations, versioning, and integrity checks. **Why Checkpoint sharding Matters** - **Parallel I/O**: Multiple writers reduce checkpoint wall-clock time on distributed storage. - **Scalability**: Supports trillion-parameter class states and multi-node optimizer partitioning. - **Failure Isolation**: Shard-level retries can recover partial write failures without restarting full save. - **Storage Throughput**: Better aligns with striped or object-based storage architectures. - **Operational Flexibility**: Shards can be replicated or migrated independently by policy. **How It Is Used in Practice** - **Shard Strategy**: Partition by rank and tensor groups to balance shard size and restore complexity. - **Manifest Management**: Persist atomic index metadata containing shard checksums and topology info. - **Restore Drills**: Regularly test multi-shard recovery under node-loss and partial-corruption scenarios. Checkpoint sharding is **the standard reliability pattern for large distributed model states** - parallel shard persistence enables scalable save and recovery at modern training sizes.