Fault tolerance is the ability to preserve defined correctness and service behavior when components fail. Distributed training, inference and data systems inevitably encounter device, process, network, storage and software faults, so recovery semantics must be designed rather than improvised. Tolerance may mask failure through replication, recover from durable checkpoints, reconstruct data with erasure codes, retry idempotent operations, isolate dependencies with circuit breakers, or reconfigure membership through consensus. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. State the fault model—crash, omission, timing, corruption or Byzantine—plus detectable symptoms, tolerated count, consistency, degraded behavior, recovery, data loss and verification.
Architecture, control plane, and operating behavior. Hardware uses ECC, redundant power and links; processes use supervisors and replicas; distributed state uses quorum and logs; training stores model, optimizer, scheduler, RNG and data position; storage uses replication or erasure coding across failure domains. Detect with heartbeats and checksums, fence failed ownership, elect or select replacement, restore or reconstruct state, replay idempotently, reconcile outputs, warm caches, verify health and rejoin carefully. Checkpoint cadence trades steady overhead against lost work. Active replication, primary-backup, checkpoint/restart, event sourcing, consensus, erasure coding, retry/circuit breaker, graceful degradation and elastic training protect different state and failure types. The operational stack spans clients and producers, APIs or ingestion, queues and schedulers, stateless and stateful compute, accelerators, memory and storage, network fabrics, identity and policy, artifact registries, observability, automation, and human operations. Control-plane decisions and data-plane work are separated so overload or compromise in one does not silently corrupt the other. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone.
Implementation, infrastructure, and failure modes. Make writes idempotent, attach sequence and fencing tokens, checksum artifacts, use atomic manifests, place replicas across domains, stagger checkpoints, retain several generations, test elastic rank changes and bound retries. ECC corrects limited memory errors; RAID/erasure codes protect media; redundant NICs and switches protect paths; spare GPUs enable job repair. Common power, firmware, rack and control planes remain correlated risks. Split brain, stale leaders, poisoned checkpoints, simultaneous replica loss, retry storms, inconsistent data-loader resume, nondeterministic replay, checkpoint congestion and silent corruption defeat nominal redundancy. Implementation favors immutable artifacts, declarative configuration, typed schemas, idempotent operations, bounded retries with jitter, deadlines, backpressure, health and readiness probes, least privilege, encrypted transport and storage, progressive rollout, reproducible environments, and complete telemetry. Automation has dry-run, approval, audit, and rollback paths. AI infrastructure joins CPUs, GPUs or NPUs, HBM, host memory, NICs and DPUs, PCIe and scale-up links, leaf-spine networks, local and shared storage, power delivery, and cooling. Topology, NUMA locality, bandwidth, failure domains, thermal headroom, and accelerator memory determine delivered behavior and must be visible to schedulers. Common failures include retry storms, queue collapse, stale health signals, split brain, partial writes, incompatible schemas, silent data corruption, time skew, dependency amplification, capacity fragmentation, noisy neighbors, credential leakage, unbounded state, monitoring blind spots, and recovery procedures that exist only on paper. A healthy component does not prove a healthy user journey.
Verification, security, and lifecycle controls. Inject process, GPU, NIC, switch, storage and node failures; corrupt checkpoints; delay messages; partition networks; test repeated failures, rollback and reconciliation; compare resumed training and served outputs. Failure detection, recovery time, lost work, data loss, checkpoint overhead, mean time between failure, repair rate, degraded capacity, retry load, integrity errors and cost matter. Classify protected state, retention, encryption, restore authority, incident escalation, evidence and risk acceptance. Replicas are not backups when corruption propagates. Verification combines unit, contract and property tests, schema compatibility, load and soak tests, chaos and fault injection, security review, backup restoration, failover and rollback drills, dependency degradation, regional evacuation where applicable, data reconciliation, shadow traffic, canaries, and end-to-end synthetic checks. Tests run against production-like scale and permissions. Source, data, configuration, environment, model, registry metadata, infrastructure definition, dependency, image, driver, firmware, deployment, experiment, approval, incident, and rollback artifacts remain linked. Continuous controls detect drift, expired credentials, unowned resources, stale backups, regressions, policy exceptions, and unsupported versions. Owners define access, segregation of duties, data classification, residency, retention and deletion, vendor and supply-chain review, incident severity, communications, audit evidence, RTO/RPO or SLO exceptions, cost attribution, and change authority. Sensitive model and experiment artifacts receive the same integrity and confidentiality controls as source and production data.
| Technique | Protected failure | Recovery behavior | Strength | Trade-off |
|---|---|---|---|---|
| Replication | Instance/node loss | Serve from replica | Fast continuity | Capacity and consistency cost |
| Checkpoint/restart | Process/job loss | Reload durable state | Fits long training | Lost work/I/O overhead |
| Consensus log | Leader/metadata loss | Elect and replay | Authoritative state | Quorum latency/complexity |
| Erasure coding | Disk/site fragment loss | Reconstruct shards | Storage efficiency | Repair compute/network |
| Retry/circuit breaker | Transient dependency fault | Repeat or isolate | Simple local resilience | Duplication/storm risk |
| Graceful degradation | Feature/capacity loss | Reduced service | Protects core journey | Requires product semantics |
<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="Segoe UI,Arial,sans-serif"><rect width="760" height="470" fill="#0d1117"/><defs><marker id="ab" markerWidth="8" markerHeight="8" refX="6.5" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#60a5fa"/></marker><marker id="ag" markerWidth="8" markerHeight="8" refX="6.5" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#34d399"/></marker><marker id="ar" markerWidth="8" markerHeight="8" refX="6.5" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#f87171"/></marker><marker id="ah" markerWidth="8" markerHeight="8" refX="6.5" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#8b949e"/></marker></defs><text x="20" y="30" fill="#e6edf3" font-size="19" font-weight="700">Fault tolerance: checkpoint, detect, and recover</text><text x="20" y="50" fill="#8b949e" font-size="12.5">A verified checkpoint caps how much work is lost (RPO); detection, restore and replay cap how long recovery takes (RTO).</text><rect x="20" y="66" width="226" height="298" rx="7" fill="#0c141d" stroke="#30363d"/><text x="32" y="87" fill="#e6edf3" font-size="13.5" font-weight="600">Recovery timeline</text><rect x="267" y="66" width="226" height="298" rx="7" fill="#0c141d" stroke="#30363d"/><text x="279" y="87" fill="#e6edf3" font-size="13.5" font-weight="600">Redundancy & detection</text><rect x="514" y="66" width="226" height="298" rx="7" fill="#0c141d" stroke="#30363d"/><text x="526" y="87" fill="#e6edf3" font-size="13.5" font-weight="600">Checkpoint & replay</text><path d="M38,144 L110,144 L112,182 L198,182 L206,144 L230,144" fill="none" stroke="#34d399" stroke-width="2"/><line x1="38" y1="194" x2="232" y2="194" stroke="#8b949e" stroke-width="1"/><text x="232" y="206" fill="#8b949e" font-size="7.4" text-anchor="end">time →</text><text x="36" y="140" fill="#8b949e" font-size="7.4">throughput</text><line x1="70" y1="144" x2="70" y2="196" stroke="#c4b5fd" stroke-width="0.8" stroke-dasharray="2 2"/><line x1="112" y1="144" x2="112" y2="196" stroke="#f87171" stroke-width="0.8" stroke-dasharray="2 2"/><line x1="134" y1="144" x2="134" y2="196" stroke="#fbbf24" stroke-width="0.8" stroke-dasharray="2 2"/><line x1="206" y1="144" x2="206" y2="196" stroke="#34d399" stroke-width="0.8" stroke-dasharray="2 2"/><circle cx="70" cy="144" r="3" fill="#c4b5fd"/><text x="70" y="138" fill="#c4b5fd" font-size="7" text-anchor="middle">ckpt N</text><circle cx="112" cy="182" r="3" fill="#f87171"/><text x="114" y="138" fill="#f87171" font-size="7" text-anchor="middle">failure</text><text x="159" y="177" fill="#fbbf24" font-size="7" text-anchor="middle">detect + restore</text><text x="206" y="138" fill="#34d399" font-size="7" text-anchor="middle">resumed</text><line x1="70" y1="210" x2="112" y2="210" stroke="#f87171" stroke-width="1" marker-start="url(#ah)" marker-end="url(#ah)"/><text x="91" y="221" fill="#f87171" font-size="7.4" text-anchor="middle">RPO (work lost)</text><line x1="112" y1="232" x2="206" y2="232" stroke="#fbbf24" stroke-width="1" marker-start="url(#ah)" marker-end="url(#ah)"/><text x="159" y="243" fill="#fbbf24" font-size="7.4" text-anchor="middle">RTO (downtime)</text><text x="38" y="260" fill="#8b949e" font-size="7.8">Restore the latest valid state, then</text><text x="38" y="271" fill="#8b949e" font-size="7.8">replay the log to close the gap.</text><rect x="283" y="162" width="34" height="30" rx="4" fill="#12233a" stroke="#60a5fa" stroke-width="1.1"/><text x="300" y="174" fill="#93c5fd" font-size="7.4" text-anchor="middle">load</text><text x="300" y="183" fill="#93c5fd" font-size="7.4" text-anchor="middle">bal.</text><rect x="387" y="126" width="60" height="26" rx="4" fill="#153524" stroke="#34d399" stroke-width="1.1"/><text x="417" y="138" fill="#34d399" font-size="8.4" text-anchor="middle" font-weight="700">R1</text><text x="417" y="147" fill="#8b949e" font-size="6.6" text-anchor="middle">serving</text><line x1="317" y1="177" x2="385" y2="139" stroke="#8b949e" stroke-width="0.9" marker-end="url(#ah)"/><rect x="387" y="162" width="60" height="26" rx="4" fill="#3a1717" stroke="#f87171" stroke-width="1.1"/><text x="417" y="174" fill="#f87171" font-size="8.4" text-anchor="middle" font-weight="700">R2</text><text x="417" y="183" fill="#8b949e" font-size="6.6" text-anchor="middle">lost</text><line x1="317" y1="177" x2="385" y2="175" stroke="#f87171" stroke-width="0.9" marker-end="url(#ar)"/><line x1="393" y1="166" x2="441" y2="184" stroke="#f87171" stroke-width="1.6"/><line x1="441" y1="166" x2="393" y2="184" stroke="#f87171" stroke-width="1.6"/><rect x="387" y="198" width="60" height="26" rx="4" fill="#153524" stroke="#34d399" stroke-width="1.1"/><text x="417" y="210" fill="#34d399" font-size="8.4" text-anchor="middle" font-weight="700">R3</text><text x="417" y="219" fill="#8b949e" font-size="6.6" text-anchor="middle">serving</text><line x1="317" y1="177" x2="385" y2="211" stroke="#8b949e" stroke-width="0.9" marker-end="url(#ah)"/><text x="283" y="234" fill="#fbbf24" font-size="8">heartbeat timeout + quorum vote</text><text x="283" y="246" fill="#8b949e" font-size="7.8">detect the loss, then fail over.</text><rect x="281" y="256" width="198" height="32" rx="4" fill="#111a24" stroke="#30363d"/><text x="380" y="269" fill="#e6edf3" font-size="9.2" text-anchor="middle" font-weight="600">2-of-3 quorum keeps serving</text><text x="380" y="281" fill="#8b949e" font-size="8" text-anchor="middle">replicas cover the gap during repair</text><text x="526" y="108" fill="#8b949e" font-size="8.4">a checkpoint snapshots all state</text><rect x="532" y="116" width="120" height="70" rx="5" fill="#1b1030" stroke="#c4b5fd" stroke-width="1.1"/><text x="592" y="129" fill="#d6c7f5" font-size="8.4" text-anchor="middle" font-weight="700">checkpoint N</text><circle cx="542" cy="139" r="1.8" fill="#c4b5fd"/><text x="548" y="142" fill="#cdd9e5" font-size="7.6">model weights</text><circle cx="542" cy="150" r="1.8" fill="#c4b5fd"/><text x="548" y="153" fill="#cdd9e5" font-size="7.6">optimizer state</text><circle cx="542" cy="161" r="1.8" fill="#c4b5fd"/><text x="548" y="164" fill="#cdd9e5" font-size="7.6">data cursor</text><circle cx="542" cy="172" r="1.8" fill="#c4b5fd"/><text x="548" y="175" fill="#cdd9e5" font-size="7.6">RNG seed</text><line x1="592" y1="186" x2="592" y2="200" stroke="#8b949e" stroke-width="0.9" marker-end="url(#ah)"/><path d="M570,206 a22,4 0 0 1 44,0 l0,20 a22,4 0 0 1 -44,0 Z" fill="#0f2033" stroke="#4d7aa8" stroke-width="1"/><ellipse cx="592" cy="206" rx="22" ry="4" fill="#12233a" stroke="#4d7aa8" stroke-width="1"/><text x="592" y="220" fill="#93c5fd" font-size="7.4" text-anchor="middle">durable store</text><text x="664" y="124" fill="#e6edf3" font-size="8.8" font-weight="600">replay</text><text x="664" y="135" fill="#8b949e" font-size="7.2">log after ckpt</text><rect x="664" y="146" width="56" height="11" rx="2" fill="#153524" stroke="#34d399" stroke-width="0.8"/><text x="692" y="154" fill="#6ee7b7" font-size="6.8" text-anchor="middle">op 1</text><rect x="664" y="161" width="56" height="11" rx="2" fill="#153524" stroke="#34d399" stroke-width="0.8"/><text x="692" y="169" fill="#6ee7b7" font-size="6.8" text-anchor="middle">op 2</text><rect x="664" y="176" width="56" height="11" rx="2" fill="#153524" stroke="#34d399" stroke-width="0.8"/><text x="692" y="184" fill="#6ee7b7" font-size="6.8" text-anchor="middle">op 3</text><rect x="664" y="191" width="56" height="11" rx="2" fill="#153524" stroke="#34d399" stroke-width="0.8"/><text x="692" y="199" fill="#6ee7b7" font-size="6.8" text-anchor="middle">op 4</text><line x1="692" y1="206" x2="692" y2="220" stroke="#34d399" stroke-width="0.9" marker-end="url(#ag)"/><text x="692" y="232" fill="#34d399" font-size="7.4" text-anchor="middle">caught up</text><text x="526" y="266" fill="#8b949e" font-size="8.2">On restart: reload the latest verified</text><text x="526" y="277" fill="#8b949e" font-size="8.2">checkpoint, then replay the log after it.</text><rect x="20" y="384" width="226" height="70" rx="7" fill="#111a24" stroke="#30363d"/><text x="32" y="403" fill="#e6edf3" font-size="12.5" font-weight="700">RPO and RTO</text><text x="32" y="420" fill="#cdd9e5" font-size="10">A verified checkpoint bounds how much work a</text><text x="32" y="433" fill="#cdd9e5" font-size="10">failure can lose (RPO); detection, restore and</text><text x="32" y="446" fill="#cdd9e5" font-size="10">replay bound how long the outage lasts (RTO).</text><rect x="267" y="384" width="226" height="70" rx="7" fill="#111a24" stroke="#30363d"/><text x="279" y="403" fill="#e6edf3" font-size="12.5" font-weight="700">Redundancy and detection</text><text x="279" y="420" fill="#cdd9e5" font-size="10">Replicas and quorum keep serving while a node</text><text x="279" y="433" fill="#cdd9e5" font-size="10">is repaired; heartbeats and timeouts detect</text><text x="279" y="446" fill="#cdd9e5" font-size="10">the loss so the system fails over instead of</text><text x="279" y="459" fill="#cdd9e5" font-size="10">stalling.</text><rect x="514" y="384" width="226" height="70" rx="7" fill="#111a24" stroke="#30363d"/><text x="526" y="403" fill="#e6edf3" font-size="12.5" font-weight="700">Checkpoint and replay</text><text x="526" y="420" fill="#cdd9e5" font-size="10">Checkpoints save model, optimizer and data</text><text x="526" y="433" fill="#cdd9e5" font-size="10">cursor to durable storage; on restart the</text><text x="526" y="446" fill="#cdd9e5" font-size="10">system reloads the latest and replays the log</text><text x="526" y="459" fill="#cdd9e5" font-size="10">after it.</text></svg>
Selection and production application. Use replication for low-latency continuity, checkpoints for expensive recomputable state, erasure coding for durable large objects, consensus for authoritative metadata and graceful degradation for noncritical features. AI training, inference, storage, databases, schedulers, queues, control planes and networks require fault tolerance. Tolerance spans algorithms, checkpoints, replicas, network, storage, scheduler, security and operator recovery rather than one retry library. The useful optimization and reliability boundary is the complete user-facing system. Improving a model server, network, registry, deployment controller, or pipeline stage can move the bottleneck or weaken consistency, safety, recoverability, and cost elsewhere, so decisions are validated end to end. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.