checkpoint restart fault tolerance

**Checkpoint/Restart and Fault Tolerance** enable **resilience against hardware failures in long-running HPC simulations through periodic application state snapshots, essential for exascale computing where mean-time-between-failures measured in hours.** **System-Level vs Application-Level Checkpointing** - **System-Level (DMTCP)**: Dungeon Master of the Totality of Cells library. Captures entire process state (memory, open files, network sockets). Transparent to application. - **Advantages of System-Level**: No code modification required. Works with legacy applications. Automatic recovery without application awareness. - **Disadvantages**: Large checkpoint size (all memory including unused pages). Slower than selective application checkpoint. - **Application-Level Checkpointing**: Application explicitly saves necessary state (not entire memory). Selective, optimized checkpoints. Requires code instrumentation. **SCR (Scalable Checkpoint/Restart) Library** - **SCR Design**: Leverages node-local storage (NVMe, RAID) for fast checkpoints. Metadata distributed across multiple nodes for reliability. - **Checkpoint Stages**: Write checkpoint locally (fast, 100 GB/s NVMe). Metadata distributed synchronously (ensures recovery info available). - **Recovery**: Upon failure, restart from latest checkpoint on local storage or rebuilt from distributed metadata. - **Scalability**: Checkpoint time ~1-10 seconds for 100k node systems (vs minutes for single shared storage). Enables frequent checkpoint (every 100-1000 iterations). **Checkpoint Interval Optimization (Young's Formula)** - **Optimal Interval**: T_checkpoint = sqrt(2 × MTBF × T_checkpoint_time). Balances checkpoint overhead vs recovery loss. - **MTBF (Mean Time Between Failure)**: System reliability metric. Decreases with scale. 100k nodes, 24hr MTBF typical (T_checkpoint ~10 minutes). - **Young's Formula Derivation**: Minimize: (work lost in failure + checkpoint time) / total work. Optimal checkpoint interval sqrt(2 × MTBF × Tchkpt). - **Practical Example**: MTBF = 1 day, checkpoint time = 10 min. Optimal interval = sqrt(2 × 86400 × 600) ≈ 32,000 sec ≈ 9 hours. Trade-off: frequent checkpoint (overhead) vs longer recovery. **HDF5 and Parallel HDF5 for Checkpoint Data** - **HDF5 (Hierarchical Data Format 5)**: Self-describing binary format. Metadata + data coexist. Ideal for large scientific datasets. - **Parallel HDF5 (pHDF5)**: Collective I/O by all MPI ranks to single file. MPIIO underneath enables all-ranks-to-single-file write in parallel. - **Checkpoint Structure**: Groups per timestep (e.g., /timestep_1000/), datasets for arrays/fields. Metadata (simulation time, iteration count) stored as HDF5 attributes. - **I/O Performance**: pHDF5 achieves 100-500 GB/s aggregate throughput on large clusters (vs 10-50 GB/s single-file write). **In-Memory Checkpointing** - **Memory-to-Memory Checkpoints**: Store checkpoints in node-local DRAM (redundant copy) instead of persistent storage. Faster than disk I/O. - **Redundancy Pattern**: Checkpoint of rank R stored on different rank (cross-node mirroring). Single node failure → recover from mirror. - **Trade-off**: Extra memory overhead (~2× for mirroring). Resilience to single node failure only; multiple simultaneous failures unrecoverable. - **Hybrid**: Disk checkpoint every N iterations (slow but persistent), in-memory checkpoints between disk checkpoints (fast). Best of both. **Silent Data Corruption (SDC) Detection** - **Silent Error Threat**: Bit flips from cosmic rays, manufacturing defects undetected. Undetected errors cause incorrect results (scientific validity compromised). - **Detection via Redundancy**: Dual computation (compute twice, compare results). Bit flips detected via mismatch. Redundancy overhead ~100% (2x compute). - **Application-Level Detection**: Sanity checks on computed quantities. Example: conservation laws (energy, mass), bounds checks (physical reasonableness). - **Lightweight Checking**: Checksum computation (CRC over data). Detects most errors, overhead ~10% (single checksum pass). **Exascale Fault Tolerance Challenges** - **Failure Rate Scaling**: Mean-time-between-failure inversely proportional to system size. 1 exaflop = 10^18 FLOP/s = 1000 PF systems (combined). MTBF ~30 minutes at exascale. - **Checkpoint Overhead**: Checkpoint time scales with data size. Exascale system checkpoint ~100GB-1TB (Checkpoint time = 10-100 sec @ 10TB/s interconnect). - **Optimal Interval**: sqrt(2 × 1800s × 50s) ≈ 450 seconds. Checkpoint every ~7 minutes optimal. Overhead: 50/450 ≈ 11% lost to checkpointing. - **Novel Approaches**: Algorithmic redundancy (encode computation, tolerate errors), lossy compression (lower checkpoint precision for reduced size), in-situ analytics (checkpoint only critical outputs). **Recovery Mechanisms and Rollback** - **Checkpoint-Restart Workflow**: Upon failure detected, job stopped, latest checkpoint loaded, simulation resumed from checkpoint. - **Rollback Logic**: Simulation time set to checkpoint timestamp. Iteration counters reset. Environment variables, pseudo-random state restored. - **Data Validity**: Post-recovery, simulation continues as if failure never occurred. Transparent recovery (from application perspective). - **Checkpoint Frequency Trade-off**: Frequent checkpoint = low rollback loss, high overhead. Sparse checkpoint = low overhead, high rollback loss. Optimized balance.