automatic recovery

**Automatic recovery** is the **self-managed restart and continuation workflow triggered by training or infrastructure failure events** - it removes manual pager-based intervention and shortens downtime after faults. **What Is Automatic recovery?** - **Definition**: System capability to detect failure, relaunch workload, and resume from valid state automatically. - **Trigger Sources**: Heartbeat timeouts, process exit codes, scheduler eviction signals, and health-check failures. - **Recovery Inputs**: Latest durable checkpoint, job specification, and resource availability rules. - **Control Logic**: Includes retry limits, exponential backoff, and escalation policies. **Why Automatic recovery Matters** - **Downtime Reduction**: Automated recovery starts immediately, minimizing idle cluster time. - **Labor Efficiency**: Fewer manual restarts reduce operator burden and off-hours interruptions. - **Consistency**: Standardized recovery workflow avoids ad hoc human error during incidents. - **Higher Completion Rates**: Long training jobs survive transient infrastructure faults more often. - **Operational Maturity**: Self-healing behavior is core to dependable platform service levels. **How It Is Used in Practice** - **Watchdog Integration**: Use orchestration controllers to monitor liveness and enforce restart policy. - **Checkpoint Selection**: Automate restore from last confirmed checkpoint with integrity validation. - **Escalation Design**: Route repeated-failure cases to humans only after policy thresholds are exceeded. Automatic recovery is **a critical self-healing capability for production ML infrastructure** - fast autonomous restart keeps training pipelines resilient and predictable.