HPC Job Scheduling (SLURM, PBS) is the cluster resource management and workload scheduling system that allocates compute nodes, GPUs, memory, and time to user-submitted batch jobs — enabling fair, efficient sharing of expensive HPC and AI training clusters among hundreds of users by queuing jobs, enforcing resource limits, managing priorities, and automating node allocation, with SLURM being the dominant scheduler powering most of the world's top supercomputers and AI training clusters.

Why Job Schedulers Are Needed

• HPC clusters: 100-10,000+ nodes, $10M-$1B+ investment.
• Multiple users/projects competing for resources.
• Without scheduler: Users fight for nodes, waste resources, no fairness.
• With scheduler: Automated allocation, queuing, priority, accounting.

Major Schedulers

SLURM Job Submission

#!/bin/bash
#SBATCH --job-name=train_llm
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --mem=1024G
#SBATCH --time=72:00:00
#SBATCH --output=train_%j.log

module load cuda/12.2 nccl/2.18
srun torchrun --nproc_per_node=8 --nnodes=4 train.py

# Submit job
sbatch train_job.sh

# Check queue
squeue -u $USER

# Cancel job
scancel 12345

# Check cluster status
sinfo

SLURM Architecture

• slurmctld: Central controller daemon — manages queue, scheduling decisions.
• slurmd: Node daemon — runs on each compute node, launches/monitors jobs.
• slurmdbd: Database daemon — stores accounting, job history.
• srun: Launch parallel tasks within an allocation.
• sbatch: Submit batch job scripts.
• salloc: Interactive allocation request.

Scheduling Policies

GPU Scheduling with SLURM

• GRES (Generic Resources): SLURM tracks GPUs as generic resources.
• Allocation: `--gpus-per-node=8` reserves 8 GPUs per node.
• GPU binding: SLURM sets CUDA_VISIBLE_DEVICES automatically.
• MIG (Multi-Instance GPU): SLURM can allocate MIG instances as separate resources.
• Topology-aware: SLURM can consider NVLink topology when co-locating tasks.

AI Training Cluster Patterns

• Large-scale training: Reserve entire nodes exclusively (no sharing).
• Inference serving: Pack multiple inference jobs per node (GPU sharing).
• Checkpointing: Jobs save state periodically → can be preempted and restarted.
• Elastic training: Jobs can scale up/down as resources become available.

HPC job scheduling is the operating system of supercomputing — SLURM and similar schedulers transform a collection of individual compute nodes into a coherent, shared computing resource that can run thousands of concurrent jobs while ensuring fair access, efficient utilization, and the multi-node coordination essential for training modern AI models that span hundreds of GPUs.