gang scheduling

**Gang scheduling** is the **allocation model requiring all requested resources for a distributed job to be available simultaneously before start** - it avoids deadlock and partial-allocation failures in tightly coupled multi-GPU training workloads. **What Is Gang scheduling?** - **Definition**: All-or-nothing launch policy for jobs that need complete synchronized resource sets. - **Distributed Need**: Many training frameworks cannot progress if only a subset of ranks starts. - **Queue Behavior**: Large gang jobs may wait longer but run correctly once admitted. - **Failure Prevention**: Prevents resource fragmentation and partial-start deadlocks. **Why Gang scheduling Matters** - **Correctness**: Ensures distributed process groups initialize with full world size as required. - **Scheduler Efficiency**: Avoids wasting resources on jobs that cannot make progress. - **Operational Reliability**: Reduces partial allocation churn and repeated launch failures. - **Predictable Runtime**: Once started, gang jobs are less likely to stall due to missing peers. - **Scale Readiness**: Essential for stable operation of large synchronized training jobs. **How It Is Used in Practice** - **Admission Control**: Reserve full requested resource bundle before signaling job start. - **Fragmentation Management**: Use backfill and packing strategies to reduce long wait times for gang jobs. - **Queue Policy**: Separate large gang queues from small jobs to improve overall fairness and utilization. Gang scheduling is **a necessary orchestration policy for synchronized distributed training** - full-cohort admission prevents partial-start failures and improves large-job reliability.