checkpoint
**Checkpoint is a durable snapshot of model and optimizer training state used for restart, selection, transfer, and reproducibility.** At large scale, hardware failures, preemption, software crashes, and planned maintenance are expected, so checkpoint design directly affects useful cluster time, storage cost, and whether training can resume correctly. A resumable checkpoint may include parameters, master weights, optimizer moments, scheduler step, gradient scaler, RNG states, data-loader position, tokenizer/config, parallel-shard metadata, and training counters. A weights-only export is not equivalent to a training checkpoint. A professional system definition specifies the data and model version, numerical precision, batch and sequence shape, parallel topology, storage and network assumptions, target accelerators, failure model, reproducibility boundary, and end-to-end objective. Isolated kernel throughput or one benchmark does not describe delivered training or retrieval behavior.
**Architecture, representation, and operating mechanism.** Full checkpoints write all state, sharded checkpoints let ranks write partitions, asynchronous checkpoints copy state to host or staging storage before background persistence, incremental/differential methods store changes, and distributed checkpoint formats support resharding to a new topology. A coordinator selects a consistent step, quiesces or snapshots state, ranks write temporary objects with checksums, a manifest commits atomically after all pieces succeed, retention policy promotes or deletes versions, and restart verifies artifacts before reconstructing ranks and data position. Snapshot wall time and training pause, effective write bandwidth, checkpoint size, storage amplification, frequency, expected lost work, async overlap, host-memory staging, restore and reshard time, integrity failure, recovery success, retention cost, and best-model utility matter. Accelerators, CPUs, HBM, host RAM, storage, interconnect, schedulers, containers, libraries, compilers, telemetry, registries, APIs, security policy, and operators form one system. Optimizing one stage can move the bottleneck or weaken correctness, isolation, and recoverability. Evaluation reports quality together with throughput, tail latency, accelerator utilization, HBM and host memory, communication volume, storage bandwidth, checkpoint or index cost, energy, fault recovery, scalability, and total cost. Controlled baselines hold data, optimization, hardware, and evaluation constant so an infrastructure change is not confused with extra compute or information.
**Implementation, infrastructure, and failure modes.** Tensor sharding and safetensor-like containers, multipart object writes, burst buffers, compression where useful, deduplication, incremental chunks, two-phase manifests, checksums, erasure/replication, async threads/processes, preemption signals, and garbage collection build robust saves. A 100B+-parameter model plus optimizer can reach terabytes. Aggregate GPU-to-host, PCIe/NVLink, DRAM, NIC, filesystem/object-store bandwidth, metadata operations, and rack contention determine pause; staging may compete with data loading and collectives. Partial files appear valid, ranks save different steps, optimizer or RNG is omitted, async buffers are overwritten, storage throttles all jobs, topology-specific shards cannot restore, data replay/skip changes optimization, corrupted old versions are discovered only after a crash, and retention deletes the last good state. Engineering includes data movement, finite precision, concurrency, resource contention, security boundaries, error propagation, and deterministic behavior when assumptions fail. Data ingestion, preprocessing, training or indexing, evaluation, artifact registration, deployment, monitoring, refresh, rollback, retention, and deletion form one lifecycle. Dataset, tokenizer, code, dependency, seed, configuration, compiler, kernel, checkpoint, index, prompt, and hardware topology versions remain linked for reproducibility and audit.
**Evaluation, governance, and deployment.** Automate save-resume equivalence, interrupt at every phase, corrupt/miss shards, change world size/topology, restore on fresh nodes, compare next losses/updates, verify data position and RNG, stress concurrent jobs, measure pause/tail, and periodically perform disaster restore. Trainer, distributed framework, scheduler, preemption, local NVMe, parallel filesystem/object store, metadata DB, experiment registry, model evaluation, artifact promotion, security, and retention form checkpoint operations. Checkpoints contain valuable IP and may memorize sensitive data. Encryption, least privilege, tenant isolation, provenance, regional storage, retention, legal hold, deletion, export controls, signing, and audit apply to every replica. Verification combines unit and property tests, numerical references, distributed fault injection, determinism checks, scale tests, performance traces, data-leakage audits, corruption recovery, hardware-in-loop measurement, offline task evaluation, shadow traffic, and canary rollout. Failures are reproducible from immutable artifacts rather than inferred from dashboards. Data ingestion, preprocessing, training or indexing, evaluation, artifact registration, deployment, monitoring, refresh, rollback, retention, and deletion form one lifecycle. Dataset, tokenizer, code, dependency, seed, configuration, compiler, kernel, checkpoint, index, prompt, and hardware topology versions remain linked for reproducibility and audit. Evaluation reports quality together with throughput, tail latency, accelerator utilization, HBM and host memory, communication volume, storage bandwidth, checkpoint or index cost, energy, fault recovery, scalability, and total cost. Controlled baselines hold data, optimization, hardware, and evaluation constant so an infrastructure change is not confused with extra compute or information.
| Strategy | Written state | Training pause | Strength | Trade-off |
|---|---|---|---|---|
| Full synchronous | Complete state each save | High at scale | Simple independent restore | Bandwidth/storage |
| Sharded distributed | Per-rank state shards | Parallel/medium | Scales aggregate writes | Manifest/topology complexity |
| Asynchronous | Host/staging then background | Low visible pause | Overlaps I/O | Extra memory/consistency |
| Incremental/differential | Changed chunks | Low-medium | Lower storage traffic | Dependency chain/restore |
| Weights-only | Model parameters | Low | Serving/selection artifact | Cannot exactly resume training |
```svg
```
**Selection and practical application.** Use sharded distributed saves for scale, asynchronous staging when pause dominates and memory permits, incremental methods when state changes/storage justify complexity, and frequent lightweight weights snapshots only when optimizer-perfect resume is unnecessary. Pretraining, fine-tuning, hyperparameter search, spot/preemptible jobs, elastic clusters, model selection, rollback, transfer, and long scientific runs rely on checkpoints. Accelerators, CPUs, HBM, host RAM, storage, interconnect, schedulers, containers, libraries, compilers, telemetry, registries, APIs, security policy, and operators form one system. Optimizing one stage can move the bottleneck or weaken correctness, isolation, and recoverability. A professional system definition specifies the data and model version, numerical precision, batch and sequence shape, parallel topology, storage and network assumptions, target accelerators, failure model, reproducibility boundary, and end-to-end objective. Isolated kernel throughput or one benchmark does not describe delivered training or retrieval behavior. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.