High availability designs a service to remain usable through routine failures, maintenance and demand changes within a stated availability objective. AI serving needs redundant model replicas, healthy dependencies and graceful degradation so one accelerator, node or region does not become user-visible downtime. Annual downtime is approximately 8.76 hours at 99.9 percent, 52.6 minutes at 99.99 percent and 5.26 minutes at 99.999 percent, but short-window and per-request SLOs often matter more than annual arithmetic. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Define good event, measurement point, window, exclusions, dependency budget, maintenance treatment, regional scope, latency and quality thresholds, and degraded-service policy.

Architecture, control plane, and operating behavior. Active-passive keeps a promoted standby; active-active serves from multiple replicas or sites; health probes and service discovery remove failures; autoscaling supplies capacity; quorum protects state; multi-region designs route around site loss. Continuously observe synthetic and real traffic, compare error budget, drain maintenance, fail over on confirmed health, shift traffic gradually, preserve session/state, shed optional work and restore normal redundancy after repair. Single-zone redundancy, multi-zone, regional active-passive, multi-region active-active, cell architecture and provider diversity offer rising isolation with greater state and operational complexity. The operational stack spans clients and producers, APIs or ingestion, queues and schedulers, stateless and stateful compute, accelerators, memory and storage, network fabrics, identity and policy, artifact registries, observability, automation, and human operations. Control-plane decisions and data-plane work are separated so overload or compromise in one does not silently corrupt the other. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone.

Implementation, infrastructure, and failure modes. Eliminate single points, use independent failure domains, readiness probes, rolling maintenance, tested failover, spare capacity, dependency timeouts, queues, admission control, stateless frontends and replicated state. Redundant power, NICs, switches, racks and sites matter, while shared storage, DNS, identity, certificate, model registry and control-plane dependencies can remain hidden single points. Health false positives trigger oscillation, state lags during failover, all replicas share a bad release, multi-region writes split, overload follows traffic shift, DNS caches delay recovery and standby rot goes unnoticed. Implementation favors immutable artifacts, declarative configuration, typed schemas, idempotent operations, bounded retries with jitter, deadlines, backpressure, health and readiness probes, least privilege, encrypted transport and storage, progressive rollout, reproducible environments, and complete telemetry. Automation has dry-run, approval, audit, and rollback paths. AI infrastructure joins CPUs, GPUs or NPUs, HBM, host memory, NICs and DPUs, PCIe and scale-up links, leaf-spine networks, local and shared storage, power delivery, and cooling. Topology, NUMA locality, bandwidth, failure domains, thermal headroom, and accelerator memory determine delivered behavior and must be visible to schedulers. Common failures include retry storms, queue collapse, stale health signals, split brain, partial writes, incompatible schemas, silent data corruption, time skew, dependency amplification, capacity fragmentation, noisy neighbors, credential leakage, unbounded state, monitoring blind spots, and recovery procedures that exist only on paper. A healthy component does not prove a healthy user journey.

Verification, security, and lifecycle controls. Exercise instance, zone, region, dependency and control-plane failure; maintain load during failover; test bad deployments, expired credentials and network partitions; measure user-visible recovery and correctness. Availability, successful goodput, p99 latency, error budget burn, detection/failover time, degraded duration, replica health, capacity headroom, data consistency and operator toil matter. SLOs align product and engineering priorities; exceptions, maintenance, incident communication, escalation, compliance, postmortems and budget decisions need accountable owners. Verification combines unit, contract and property tests, schema compatibility, load and soak tests, chaos and fault injection, security review, backup restoration, failover and rollback drills, dependency degradation, regional evacuation where applicable, data reconciliation, shadow traffic, canaries, and end-to-end synthetic checks. Tests run against production-like scale and permissions. Source, data, configuration, environment, model, registry metadata, infrastructure definition, dependency, image, driver, firmware, deployment, experiment, approval, incident, and rollback artifacts remain linked. Continuous controls detect drift, expired credentials, unowned resources, stale backups, regressions, policy exceptions, and unsupported versions. Owners define access, segregation of duties, data classification, residency, retention and deletion, vendor and supply-chain review, incident severity, communications, audit evidence, RTO/RPO or SLO exceptions, cost attribution, and change authority. Sensitive model and experiment artifacts receive the same integrity and confidentiality controls as source and production data.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif"><rect width="760" height="470" fill="#0d1117"/><defs><marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10Z" fill="#60a5fa"/></marker><filter id="glow"><feGaussianBlur stdDeviation="7"/></filter></defs><text x="380" y="34" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">High Availability — Fail Over Across Zones</text><text x="380" y="56" fill="#8b98a5" font-size="13" text-anchor="middle">health checks stop routing to a failed replica while synchronized capacity remains available</text><g transform="translate(45 190)"><circle cx="30" cy="30" r="25" fill="#172033" stroke="#93c5fd"/><path d="M13 37q17-26 34 0M18 19h24" fill="none" stroke="#93c5fd" stroke-width="2"/><text x="30" y="70" fill="#8b98a5" font-size="11" text-anchor="middle">requests</text></g><path d="M108 220H177" stroke="#60a5fa" stroke-width="2.5" marker-end="url(#arrow)"/><g transform="translate(185 178)"><path d="M0 42L48 0L96 42L48 84Z" fill="#171421" stroke="#a78bfa" stroke-width="2"/><text x="48" y="37" fill="#c4b5fd" font-size="10" font-weight="700" text-anchor="middle">health-aware</text><text x="48" y="53" fill="#c4b5fd" font-size="10" text-anchor="middle">load balancer</text></g><rect x="340" y="88" width="350" height="135" rx="9" fill="#0b1726" stroke="#60a5fa"/><text x="357" y="110" fill="#93c5fd" font-size="12" font-weight="700">ZONE A</text><rect x="370" y="130" width="105" height="58" rx="7" fill="#1e3a5f" stroke="#60a5fa"/><text x="422" y="164" fill="#bfdbfe" font-size="11" text-anchor="middle">service A</text><ellipse cx="610" cy="159" rx="55" ry="29" fill="#172033" stroke="#60a5fa"/><text x="610" y="163" fill="#bfdbfe" font-size="11" text-anchor="middle">primary DB</text><path d="M476 159H548" stroke="#60a5fa" stroke-width="2" marker-end="url(#arrow)"/><rect x="340" y="262" width="350" height="135" rx="9" fill="#0d201c" stroke="#34d399"/><text x="357" y="284" fill="#6ee7b7" font-size="12" font-weight="700">ZONE B</text><rect x="370" y="304" width="105" height="58" rx="7" fill="#164e3d" stroke="#34d399"/><text x="422" y="338" fill="#a7f3d0" font-size="11" text-anchor="middle">service B</text><ellipse cx="610" cy="333" rx="55" ry="29" fill="#15332b" stroke="#34d399"/><text x="610" y="337" fill="#a7f3d0" font-size="11" text-anchor="middle">standby DB</text><path d="M476 333H548" stroke="#34d399" stroke-width="2" marker-end="url(#arrow)"/><path d="M281 205C310 205 311 159 340 159" fill="none" stroke="#f87171" stroke-width="2" stroke-dasharray="5 4"/><path d="M281 235C310 235 311 333 340 333" fill="none" stroke="#34d399" stroke-width="3" marker-end="url(#arrow)"/><g stroke="#f87171" stroke-width="4"><path d="M389 138l66 42M455 138l-66 42"/></g><text x="422" y="202" fill="#fca5a5" font-size="10" text-anchor="middle">unhealthy → removed</text><path d="M610 190V299" stroke="#a78bfa" stroke-width="2" stroke-dasharray="6 4" marker-end="url(#arrow)"/><text x="626" y="248" fill="#c4b5fd" font-size="10">replication</text><path d="M555 333C515 333 515 232 555 232" fill="none" stroke="#fbbf24" stroke-width="2" marker-end="url(#arrow)"/><text x="505" y="253" fill="#fbbf24" font-size="10" text-anchor="middle">promote</text><text x="380" y="452" fill="#6b7684" font-size="11.5" text-anchor="middle">Availability comes from independent failure domains, spare capacity, tested failover, and bounded data loss.</text></svg>

Selection and production application. Use multi-zone active-active for common services, active-passive where state or cost dominates, multi-region for justified business impact, and five-nines targets only when architecture and operations support them. Inference APIs, registries, feature stores, pipelines, control planes, databases and customer applications use HA patterns. Availability is the product of dependencies, capacity, releases, data, network, identity, model behavior and operations—not replica count alone. The useful optimization and reliability boundary is the complete user-facing system. Improving a model server, network, registry, deployment controller, or pipeline stage can move the bottleneck or weaken consistency, safety, recoverability, and cost elsewhere, so decisions are validated end to end. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.