Disaster recovery restores critical services and data after catastrophic site loss, widespread corruption, destructive attack or prolonged control-plane failure. High availability handles local faults; disaster recovery addresses events large enough to invalidate the primary environment and its normal redundancy. Recovery time objective is the targeted maximum time to restore a business capability; recovery point objective is the maximum acceptable data loss measured backward from disruption. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. A DR plan names business services, dependencies, data classes, RTO/RPO, backup immutability, site/provider failure, cyber recovery, authority, communications, validation and return-to-primary.

Architecture, control plane, and operating behavior. The primary region replicates selected data and artifacts to an isolated recovery site or account. Backups, model weights, registry metadata, checkpoints, infrastructure code, secrets recovery and DNS/routing must all be available without the failed control plane. Declare incident, contain and preserve evidence, select a clean recovery point, establish identity and network, restore control and data planes in dependency order, validate integrity and capacity, shift traffic, reconcile lost/divergent writes and later fail back safely. Backup/restore minimizes standing cost; pilot light keeps core state; warm standby keeps scaled-down service; hot or active-active sites offer shortest recovery at highest complexity; isolated cyber vaults protect clean copies. The operational stack spans clients and producers, APIs or ingestion, queues and schedulers, stateless and stateful compute, accelerators, memory and storage, network fabrics, identity and policy, artifact registries, observability, automation, and human operations. Control-plane decisions and data-plane work are separated so overload or compromise in one does not silently corrupt the other. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone.

Implementation, infrastructure, and failure modes. Use immutable versioned backups, cross-account/region replication, independent credentials, tested infrastructure code, dependency maps, restoration automation, capacity reservations, runbooks, contact trees and regular exercises. Recovery needs compatible accelerator capacity, networking, storage bandwidth, power and quotas. Models can be large, so transfer and checksum time plus scarce GPUs can dominate RTO. Backups replicate corruption, keys are unavailable, restore bandwidth is too low, DNS and certificates depend on the failed site, model/data versions mismatch, quotas block scale, and untested runbooks exceed objectives. Implementation favors immutable artifacts, declarative configuration, typed schemas, idempotent operations, bounded retries with jitter, deadlines, backpressure, health and readiness probes, least privilege, encrypted transport and storage, progressive rollout, reproducible environments, and complete telemetry. Automation has dry-run, approval, audit, and rollback paths. AI infrastructure joins CPUs, GPUs or NPUs, HBM, host memory, NICs and DPUs, PCIe and scale-up links, leaf-spine networks, local and shared storage, power delivery, and cooling. Topology, NUMA locality, bandwidth, failure domains, thermal headroom, and accelerator memory determine delivered behavior and must be visible to schedulers. Common failures include retry storms, queue collapse, stale health signals, split brain, partial writes, incompatible schemas, silent data corruption, time skew, dependency amplification, capacity fragmentation, noisy neighbors, credential leakage, unbounded state, monitoring blind spots, and recovery procedures that exist only on paper. A healthy component does not prove a healthy user journey.

Verification, security, and lifecycle controls. Restore from scratch in isolation, simulate region and identity loss, verify checksums and model quality, measure every dependency, test cyber-clean-room recovery, reconcile data and conduct full failback. Achieved RTO/RPO, backup success and age, restore throughput, integrity failures, dependency recovery, exercise frequency, capacity gap, data loss, cost and business impact matter. Business owners approve RTO/RPO, backup retention, legal hold, residency, encryption/key escrow, cyber response, crisis communications, vendor obligations and residual risk. Verification combines unit, contract and property tests, schema compatibility, load and soak tests, chaos and fault injection, security review, backup restoration, failover and rollback drills, dependency degradation, regional evacuation where applicable, data reconciliation, shadow traffic, canaries, and end-to-end synthetic checks. Tests run against production-like scale and permissions. Source, data, configuration, environment, model, registry metadata, infrastructure definition, dependency, image, driver, firmware, deployment, experiment, approval, incident, and rollback artifacts remain linked. Continuous controls detect drift, expired credentials, unowned resources, stale backups, regressions, policy exceptions, and unsupported versions. Owners define access, segregation of duties, data classification, residency, retention and deletion, vendor and supply-chain review, incident severity, communications, audit evidence, RTO/RPO or SLO exceptions, cost attribution, and change authority. Sensitive model and experiment artifacts receive the same integrity and confidentiality controls as source and production data.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif"><rect width="760" height="470" fill="#0d1117"/><defs><marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#60a5fa"/></marker><marker id="green" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#34d399"/></marker></defs><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Disaster Recovery — Restore Service from a Separate Region</text><text x="380" y="56" fill="#8b98a5" font-size="13" text-anchor="middle">replicated state, immutable backups, and rehearsed failover bound data loss and recovery time</text><g transform="translate(43 96)"><path d="M0 0h272v264H0Z" fill="#10233b" stroke="#60a5fa" stroke-width="2"/><text x="136" y="25" fill="#93c5fd" font-size="11" font-weight="700" text-anchor="middle">PRIMARY REGION</text><g transform="translate(26 52)"><rect width="105" height="64" rx="7" fill="#101a28" stroke="#60a5fa" stroke-width="2"/><g fill="#60a5fa"><circle cx="16" cy="17" r="4"/><circle cx="16" cy="32" r="4"/><circle cx="16" cy="47" r="4"/></g><path d="M29 17h59M29 32h46M29 47h54" stroke="#64748b"/><text x="52" y="81" fill="#60a5fa" font-size="9.5" text-anchor="middle">application</text></g><g transform="translate(141 52)"><rect width="105" height="64" rx="7" fill="#101a28" stroke="#60a5fa" stroke-width="2"/><g fill="#60a5fa"><circle cx="16" cy="17" r="4"/><circle cx="16" cy="32" r="4"/><circle cx="16" cy="47" r="4"/></g><path d="M29 17h59M29 32h46M29 47h54" stroke="#64748b"/><text x="52" y="81" fill="#60a5fa" font-size="9.5" text-anchor="middle">application</text></g><g transform="translate(77 157)"><path d="M0 12C0 0 118 0 118 12v58c0 12-118 12-118 0V12" fill="#101a28" stroke="#a78bfa"/><ellipse cx="59" cy="12" rx="59" ry="12" fill="#211936" stroke="#a78bfa"/><text x="59" y="50" fill="#c4b5fd" font-size="10.5" text-anchor="middle">database LSN 847</text></g></g><path d="M316 232H443" stroke="#34d399" stroke-width="3" marker-end="url(#green)"/><text x="380" y="218" fill="#6ee7b7" font-size="9.5" text-anchor="middle">async replication · RPO</text><g transform="translate(445 96)"><path d="M0 0h272v264H0Z" fill="#123c35" stroke="#34d399" stroke-width="2"/><text x="136" y="25" fill="#6ee7b7" font-size="11" font-weight="700" text-anchor="middle">RECOVERY REGION</text><g transform="translate(26 52)"><rect width="105" height="64" rx="7" fill="#101a28" stroke="#34d399" stroke-width="2"/><g fill="#34d399"><circle cx="16" cy="17" r="4"/><circle cx="16" cy="32" r="4"/><circle cx="16" cy="47" r="4"/></g><path d="M29 17h59M29 32h46M29 47h54" stroke="#64748b"/><text x="52" y="81" fill="#34d399" font-size="9.5" text-anchor="middle">warm standby</text></g><g transform="translate(141 52)"><rect width="105" height="64" rx="7" fill="#101a28" stroke="#34d399" stroke-width="2"/><g fill="#34d399"><circle cx="16" cy="17" r="4"/><circle cx="16" cy="32" r="4"/><circle cx="16" cy="47" r="4"/></g><path d="M29 17h59M29 32h46M29 47h54" stroke="#64748b"/><text x="52" y="81" fill="#34d399" font-size="9.5" text-anchor="middle">warm standby</text></g><g transform="translate(77 157)"><path d="M0 12C0 0 118 0 118 12v58c0 12-118 12-118 0V12" fill="#101a28" stroke="#34d399"/><ellipse cx="59" cy="12" rx="59" ry="12" fill="#123c35" stroke="#34d399"/><text x="59" y="50" fill="#6ee7b7" font-size="10.5" text-anchor="middle">replica LSN 842</text></g></g><g transform="translate(279 354)"><path d="M0 12C0 0 202 0 202 12v56c0 12-202 12-202 0V12" fill="#392d12" stroke="#f59e0b"/><ellipse cx="101" cy="12" rx="101" ry="12" fill="#2a2112" stroke="#f59e0b"/><text x="101" y="48" fill="#fbbf24" font-size="10.5" text-anchor="middle">immutable backup · separate credentials</text></g><path d="M203 360C219 396 252 396 279 389M557 360C541 396 508 396 481 389" fill="none" stroke="#f59e0b" stroke-width="2" marker-end="url(#arrow)"/><path d="M178 96C251 62 508 62 582 96" fill="none" stroke="#f87171" stroke-width="3" stroke-dasharray="7 5" marker-end="url(#arrow)"/><rect x="193" y="65" width="374" height="21" rx="10" fill="#0d1117"/><text x="380" y="79" fill="#fca5a5" font-size="10.5" text-anchor="middle">health authority changes DNS / traffic after declared failover · RTO</text><path d="M43 117l272 223M315 117L43 340" stroke="#f87171" stroke-width="4" opacity=".35"/><text x="380" y="452" fill="#6b7684" font-size="11.5" text-anchor="middle">Recovery is credible only when dependencies, data consistency, access, traffic switching, and restore time are tested as one runbook.</text></svg>

Selection and production application. Use backup/restore for tolerant services, pilot light or warm standby for hours-to-minutes recovery and active-active only when near-zero objectives justify state and cost complexity. Model serving, registries, datasets, checkpoints, feature stores, experiment systems, customer data and MLOps control planes require DR. DR succeeds only when data, models, metadata, identity, network, compute, infrastructure code, observability, people and vendors recover together. The useful optimization and reliability boundary is the complete user-facing system. Improving a model server, network, registry, deployment controller, or pipeline stage can move the bottleneck or weaken consistency, safety, recoverability, and cost elsewhere, so decisions are validated end to end. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.