dataset sharding

**Dataset sharding** is the **partitioning of training data into non-overlapping subsets assigned across distributed workers** - it ensures balanced workload distribution, minimizes duplication, and supports efficient parallel training execution. **What Is Dataset sharding?** - **Definition**: Splitting a dataset into shards so each worker processes a distinct portion per epoch. - **Primary Objective**: Maximize parallelism while preserving statistical representativeness across workers. - **Sharding Modes**: Static sharding, dynamic reshuffling per epoch, and locality-aware shard assignment. - **Correctness Requirement**: Each sample should be seen with intended frequency across global training. **Why Dataset sharding Matters** - **Scalable Throughput**: Proper sharding allows many workers to consume data without contention. - **Load Balance**: Even shard sizing prevents stragglers that slow synchronized training steps. - **Network Efficiency**: Locality-aware shard placement reduces remote data fetch overhead. - **Convergence Quality**: Balanced sample exposure improves gradient quality and training stability. - **Operational Simplicity**: Clear shard logic aids reproducibility and debugging in distributed jobs. **How It Is Used in Practice** - **Shard Planning**: Choose shard size and count based on worker parallelism and dataset characteristics. - **Epoch Coordination**: Synchronize shard assignment and sampler state across all ranks. - **Integrity Checks**: Validate no unintended overlap, omission, or skew in sample consumption. Dataset sharding is **a fundamental data-parallel design element for distributed training** - good shard strategy improves utilization, convergence behavior, and system efficiency.