checkpoint

Checkpointing is the practice of saving snapshots of model weights, optimizer states, learning rate schedulers, and training metadata at regular intervals during neural network training, enabling recovery from failures, comparison of training stages, and selection of the best-performing model version. In the context of large language model training — which can take weeks or months on expensive hardware — checkpointing is critical infrastructure that protects against total loss of training progress due to hardware failures, software bugs, or power outages. A complete checkpoint typically includes: model parameters (all weight tensors — the core of the checkpoint), optimizer state (for AdamW: first and second moment estimates for every parameter — approximately 2× the model size), learning rate scheduler state (current step, remaining schedule), random number generator states (for exact reproducibility), training metadata (current epoch, step, loss values, evaluated metrics), and data loader state (position in the training data for deterministic resumption). Checkpoint strategies for large models include: periodic full checkpoints (saving everything every N steps — typically every 500-2000 steps for LLM training), asynchronous checkpointing (saving in the background without pausing training — critical for large models where checkpoint save time is significant), distributed checkpointing (each device saves its shard of the model in parallel — FSDP/ZeRO sharded checkpoints), incremental checkpoints (saving only the difference from the last checkpoint), and selective checkpoints (saving only model weights without optimizer states for evaluation-only checkpoints, reducing storage by 3×). Activation checkpointing (also called gradient checkpointing) is a related but distinct concept — it trades compute for memory during training by not storing intermediate activations, recomputing them during the backward pass. This reduces memory usage by approximately √(number of layers) but increases computation by ~30%. Best practices include maintaining multiple checkpoint generations to prevent corruption from propagating, validating checkpoint integrity, and retaining checkpoints at key training milestones.