canary deployment
**Canary deployment releases a new service or model version to a small, controlled traffic slice before broader rollout.** It limits blast radius while exposing the candidate to real workloads, dependencies and user behavior that offline tests cannot fully reproduce. A common progression begins around one to five percent, pauses for evidence, then expands through stages; exact fractions and bake time follow traffic volume, risk and statistical power. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Define allocation unit, eligibility, baseline, sticky assignment, metrics and guardrails, minimum samples and time, automatic rollback, owner, exclusions, data compatibility and ramp plan.
**Architecture, control plane, and operating behavior.** A router splits traffic between stable and canary pools, deployment control manages capacity and versions, telemetry compares latency/errors/quality/business outcomes, and a controller or operator promotes, pauses or rolls back. Deploy dark or shadow checks, admit a small cohort, verify infrastructure health, compare guardrails and model outcomes, expand gradually across failure domains, stop on error-budget burn, and complete or revert while preserving audit. Canary tests release safety; A/B tests causal product variants; shadow sends copied requests without user-visible output; rolling replaces instances gradually; blue-green swaps full environments. Strategies can be combined. The operational stack spans clients and producers, APIs or ingestion, queues and schedulers, stateless and stateful compute, accelerators, memory and storage, network fabrics, identity and policy, artifact registries, observability, automation, and human operations. Control-plane decisions and data-plane work are separated so overload or compromise in one does not silently corrupt the other. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone.
**Implementation, infrastructure, and failure modes.** Use immutable artifacts, cohort stickiness, representative traffic, capacity headroom, separate launch and experiment metrics, automated rollback, schema compatibility, database expand-contract, connection draining and no shared mutable state that corrupts stable. A second model version consumes GPU HBM and cache, can fragment batching, and may need separate replicas. Low traffic underutilizes accelerators; model multiplexing or shadowing adds capacity and cost. Canary gets only easy traffic, sample is too small, stable and canary share a bad dependency, retries cross variants, delayed harm is missed, rollback cannot undo writes, or noisy metrics trigger flapping. Implementation favors immutable artifacts, declarative configuration, typed schemas, idempotent operations, bounded retries with jitter, deadlines, backpressure, health and readiness probes, least privilege, encrypted transport and storage, progressive rollout, reproducible environments, and complete telemetry. Automation has dry-run, approval, audit, and rollback paths. AI infrastructure joins CPUs, GPUs or NPUs, HBM, host memory, NICs and DPUs, PCIe and scale-up links, leaf-spine networks, local and shared storage, power delivery, and cooling. Topology, NUMA locality, bandwidth, failure domains, thermal headroom, and accelerator memory determine delivered behavior and must be visible to schedulers. Common failures include retry storms, queue collapse, stale health signals, split brain, partial writes, incompatible schemas, silent data corruption, time skew, dependency amplification, capacity fragmentation, noisy neighbors, credential leakage, unbounded state, monitoring blind spots, and recovery procedures that exist only on paper. A healthy component does not prove a healthy user journey.
**Verification, security, and lifecycle controls.** Test routing and stickiness, rollback under load, dependency and schema changes, partial failure, capacity, metric delay, model quality, subgroup impact, sequential decision rules and sustained bake periods. Error and timeout, p99 latency, saturation, quality, calibration, safety, KPI, cohort balance, sample size, confidence or posterior, rollback time and error-budget burn matter. Define launch authority, high-impact review, user consent where experimentation requires it, data handling, subgroup safeguards, audit and incident communication. Verification combines unit, contract and property tests, schema compatibility, load and soak tests, chaos and fault injection, security review, backup restoration, failover and rollback drills, dependency degradation, regional evacuation where applicable, data reconciliation, shadow traffic, canaries, and end-to-end synthetic checks. Tests run against production-like scale and permissions. Source, data, configuration, environment, model, registry metadata, infrastructure definition, dependency, image, driver, firmware, deployment, experiment, approval, incident, and rollback artifacts remain linked. Continuous controls detect drift, expired credentials, unowned resources, stale backups, regressions, policy exceptions, and unsupported versions. Owners define access, segregation of duties, data classification, residency, retention and deletion, vendor and supply-chain review, incident severity, communications, audit evidence, RTO/RPO or SLO exceptions, cost attribution, and change authority. Sensitive model and experiment artifacts receive the same integrity and confidentiality controls as source and production data.
| Strategy | Traffic pattern | Primary objective | Rollback | Main trade-off |
|---|---|---|---|---|
| Canary | Small then ramp | Limit release blast radius | Route/remove candidate | Time/measurement complexity |
| Blue-green | One full environment then switch | Zero-downtime environment swap | Switch back | Double capacity/state changes |
| Rolling | Replace instances gradually | Capacity-efficient update | Reverse rollout | Mixed versions/slower rollback |
| A/B test | Random stable cohorts | Causal variant comparison | End experiment | Needs power and ethics |
| Shadow | Copy traffic, no response | Observe candidate safely | Stop copy | No real user outcome |
```svg
```
**Selection and production application.** Use canary for operational risk, A/B for causal product effects, shadow for no-impact observation and blue-green when full-environment switching and instant rollback are preferred. Model versions, inference runtimes, APIs, kernels, data transforms, agent prompts and infrastructure releases use canaries. Canary safety depends on routing, metrics, model registry, deployment, capacity, data schemas, rollback, observability and human decision policy. The useful optimization and reliability boundary is the complete user-facing system. Improving a model server, network, registry, deployment controller, or pipeline stage can move the bottleneck or weaken consistency, safety, recoverability, and cost elsewhere, so decisions are validated end to end. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.