hpc job scheduler slurm

**HPC Job Scheduling and Workflow Management: SLURM and DAG-Based Workflows — resource allocation and execution sequencing for batch HPC jobs and complex multi-stage scientific pipelines** **SLURM (Simple Linux Utility for Resource Management)** - **Design Philosophy**: open-source, scalable to 1000s nodes, integrated into most HPC clusters - **Architecture**: controller daemon (slurmctld) on head node, compute nodes run slurmd (agent), clients submit/query via slurm tools - **Key Components**: partitions (node groups for scheduling), queues (job queues per partition), nodes (individual compute resources) - **Scalability**: controller handles 1000s jobs, supports hierarchical scheduling (tree of controllers for 10,000+ nodes) **SLURM Job Submission (sbatch)** - **Batch Script**: shell script specifies resources (``#SBATCH' directives), input/output files, command to execute - **Example**: ``sbatch --nodes=100 --tasks-per-node=1 --cpus-per-task=4 --time=01:00:00 myjob.sh' - **Job Array**: array syntax (``--array=0-99') spawns 100 independent jobs (parameter sweep) - **Dependencies**: ``--dependency=afterok:123' ensures job 123 finishes before current job starts **SLURM Parallel Launch (srun)** - **MPI Process Binding**: srun handles MPI startup (rank assignment, process placement on cores) - **CPU Binding**: ``srun --cpu-bind=sockets' pins processes to sockets (improves memory locality) - **Heterogeneous Steps**: ``srun --job-name=gpu_step --gpus=1' runs specific step with GPU allocation **SLURM Accounting and Fairshare** - **Fairshare Algorithm**: tracks resource usage (CPU-hours per user/group), prioritizes lower-usage users - **Priority Boost**: long-waiting jobs increase priority over time (starvation prevention) - **Reservation**: advance resource reservation (``scontrol create reservation'), ensures availability for high-priority jobs - **QOS (Quality of Service)**: different tiers (standard, premium, debug), different limits/priorities **PBS/Torque Job Scheduler** - **Design**: older HPC standard (predates SLURM), similar functionality, less adoption now - **qsub Command**: equivalent to sbatch (submit job), qstat (check status), qdel (delete job) - **Compatibility**: SLURM dominance reduced PBS adoption (but still used in some facilities) - **Comparison**: SLURM more feature-rich, PBS simpler but slower to evolve **Workflow Management: Nextflow** - **DSL (Domain-Specific Language)**: describe pipeline as directed acyclic graph (DAG), intuitive for scientists - **Process Definition**: workflow consists of processes (scripts/tasks), linked by channels (data flow) - **Parallelism**: automatic parallelization (fork-join) across data items, job submission to HPC cluster via backend - **Backend Flexibility**: supports SLURM, PBS, Kubernetes, cloud platforms (same workflow portable) - **Reproducibility**: frozen dependency versions (containers, Nextflow versioning), enables publication-quality reproducibility **Snakemake Workflow Framework** - **Python-Based**: rules written in Python (familiar to scientists), conditional execution, workflow inference - **Dependency Resolution**: ``snakemake' analyzes file dependencies, constructs implicit DAG, executes in parallel - **Example**: rule align_fastq reads BAM file, outputs aligned BAM, explicit dependency modeling - **Distributed Execution**: Snakemake schedules to SLURM/cloud, similar to Nextflow but Python-first **HPC Queue Priority and Scheduling** - **FIFO (First-In-First-Out)**: fairest simple scheduling, but can starve small jobs behind large jobs - **Backfill**: scheduler identifies gaps (small jobs can fit before large job completion), fills gaps (improves utilization) - **Gang Scheduling**: time-share nodes (multiple jobs on same node, swapped via preemption), increases utilization but adds latency - **Preemption**: high-priority job preempts lower-priority (saves state if possible, or kills), ensures critical work gets resources **Resource Allocation Strategies** - **Pack**: schedule jobs densely (fill nodes completely before using new node), reduces fragmentation - **Spread**: distribute across nodes (anti-pack), improves memory bandwidth but uses more nodes - **Balance**: balance between pack/spread based on workload (compute-heavy: pack, memory-heavy: spread) - **Constraint-Based**: specify required resources (CPU cores, memory, GPU, specific node features) **Heterogeneous Job Allocation** - **Multiple Resource Types**: job requests CPU + GPU + memory (e.g., 4 CPU + 1 GPU + 8 GB memory) - **Scheduling Complexity**: scheduler must find nodes with specific resource combinations, NP-hard in general - **Heuristic Solution**: greedy packing (fit largest resource requests first) - **Utilization Impact**: heterogeneity reduces bin packing efficiency (~10-20% utilization loss) **Job Dependency Management** - **afterok**: job runs after predecessor succeeds (exit code 0) - **afternotok**: job runs if predecessor fails (exit code non-zero) - **afterany**: job runs regardless of predecessor status - **DAG Support**: Nextflow/Snakemake auto-generate dependencies (no manual specification needed) **Batch vs Interactive Jobs** - **Batch (sbatch)**: job submitted to queue, executed when resources available, results written to files (asynchronous) - **Interactive (salloc)**: allocate resources, get shell prompt on compute node, immediate feedback - **Use Cases**: batch for long-running simulations (1000+ core-hours), interactive for debugging/development - **Reservation**: interactive jobs can reserve resources (``salloc --time=1:00:00'), blocks other jobs **Advance Reservation** - **Use Case**: ensure resources available for specific time window (maintenance, deadline-driven project) - **Mechanism**: ``scontrol create reservation starttime=2024-03-01T09:00:00 duration=3600 nodes=100' - **Preemption**: reserved time guaranteed (other jobs preempted if necessary) - **Cost**: reduces cluster utilization (reserved but potentially idle), justified for critical work **Job Checkpointing and Restart** - **Checkpoint**: save job state (memory, open files, execution context) to disk - **Restart**: reload state, resume execution (avoids recomputation) - **Benefit**: enables job preemption (save + restart), fault tolerance (survive crashes) - **Mechanism**: application-level (custom code) or system-level (transparent, but limited portability) **Scientific Workflow Provenance** - **Record Execution**: track which inputs → outputs, tool versions, parameters, execution environment - **Reproducibility**: re-run same pipeline (deterministic if possible), verify results match - **PROV-DM Standard**: W3C standard for provenance representation (graph of entities, activities, agents) - **Tools**: Galaxy (web-based workflow platform), Common Workflow Language (CWL) for portable workflows **Scalability of Scheduling** - **Large Clusters (10,000+ nodes)**: scheduling becomes critical bottleneck, decision latency limits throughput - **Optimization**: approximate scheduling algorithms (not NP-hard exact solutions), fast heuristics - **Distributed Scheduling**: multiple schedulers coordinate (reduces single-point bottleneck), enables elasticity **Future Directions**: AI-driven scheduling (predict job characteristics, optimize placement), serverless HPC (FaaS model), containers standardizing job environments (reducing scheduling constraints).