blue green deployment
**Blue-green deployment maintains two production-capable environments so traffic can switch from the current blue version to a validated green version with minimal downtime.** The idle previous environment offers rapid rollback and clean release isolation, at the cost of duplicate capacity and difficult shared-state transitions. Blue serves production while green receives the new artifacts and configuration, undergoes smoke and synthetic tests, warms caches and models, then becomes the traffic target through router or DNS control. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Define environment parity, data ownership, traffic switch, session handling, warmup, health, schema compatibility, rollback window, capacity, cost, state reconciliation and decommission.
**Architecture, control plane, and operating behavior.** A stable front door points to blue or green pools; both have independent compute and deployment artifacts; shared state uses backward-compatible migrations or isolated replication; observability labels environments; control automates switch and reversal. Provision green from code, deploy immutable version, validate and warm, mirror or test traffic, freeze risky changes, switch gradually or atomically, observe, roll back if needed, then retain blue for a bounded window and rotate colors next release. Full-stack blue-green, application-only switch, regional blue-green, immutable environments and blue-green combined with a canary traffic step vary in cost and risk. Database changes need separate expand-migrate-contract workflow. The operational stack spans clients and producers, APIs or ingestion, queues and schedulers, stateless and stateful compute, accelerators, memory and storage, network fabrics, identity and policy, artifact registries, observability, automation, and human operations. Control-plane decisions and data-plane work are separated so overload or compromise in one does not silently corrupt the other. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone.
**Implementation, infrastructure, and failure modes.** Ensure parity from declarative infrastructure, deploy by digest, maintain capacity, drain connections, keep sessions external or compatible, use feature flags, make migrations backward compatible, test switch controls and limit simultaneous changes. Duplicate GPU environments can be costly and capacity-constrained; model weights, caches and kernels must warm before switch. Reserving spare accelerators reduces utilization but enables rollback. Green differs from blue infrastructure, shared migrations break old code, writes diverge, long sessions remain on blue, cold caches spike latency, DNS caches slow switch, capacity is reclaimed before confidence and rollback points to corrupted state. Implementation favors immutable artifacts, declarative configuration, typed schemas, idempotent operations, bounded retries with jitter, deadlines, backpressure, health and readiness probes, least privilege, encrypted transport and storage, progressive rollout, reproducible environments, and complete telemetry. Automation has dry-run, approval, audit, and rollback paths. AI infrastructure joins CPUs, GPUs or NPUs, HBM, host memory, NICs and DPUs, PCIe and scale-up links, leaf-spine networks, local and shared storage, power delivery, and cooling. Topology, NUMA locality, bandwidth, failure domains, thermal headroom, and accelerator memory determine delivered behavior and must be visible to schedulers. Common failures include retry storms, queue collapse, stale health signals, split brain, partial writes, incompatible schemas, silent data corruption, time skew, dependency amplification, capacity fragmentation, noisy neighbors, credential leakage, unbounded state, monitoring blind spots, and recovery procedures that exist only on paper. A healthy component does not prove a healthy user journey.
**Verification, security, and lifecycle controls.** Compare configuration and artifacts, exercise smoke/synthetic/load, warmup, full switch and switchback, session draining, schema compatibility, failure during transition, capacity, security and disaster conditions. Switch and rollback time, parity drift, errors, p99 latency, cold-start impact, connection drain, data inconsistency, duplicate capacity cost and change failure rate matter. Release and switch authority, separation of duties, change window, migration approval, audit, evidence retention and decommission authorization must be explicit. Verification combines unit, contract and property tests, schema compatibility, load and soak tests, chaos and fault injection, security review, backup restoration, failover and rollback drills, dependency degradation, regional evacuation where applicable, data reconciliation, shadow traffic, canaries, and end-to-end synthetic checks. Tests run against production-like scale and permissions. Source, data, configuration, environment, model, registry metadata, infrastructure definition, dependency, image, driver, firmware, deployment, experiment, approval, incident, and rollback artifacts remain linked. Continuous controls detect drift, expired credentials, unowned resources, stale backups, regressions, policy exceptions, and unsupported versions. Owners define access, segregation of duties, data classification, residency, retention and deletion, vendor and supply-chain review, incident severity, communications, audit evidence, RTO/RPO or SLO exceptions, cost attribution, and change authority. Sensitive model and experiment artifacts receive the same integrity and confidentiality controls as source and production data.
| Strategy | Concurrent environments | Traffic transition | Rollback speed | Primary cost/risk |
|---|---|---|---|---|
| Blue-green | Two complete | Switch after validation | Very fast routing switch | Duplicate capacity/shared state |
| Canary | Stable plus small candidate | Progressive percentage | Fast if reversible | Longer mixed operation |
| Rolling | Old/new instance mix | Instance batches | Moderate | Mixed versions/capacity |
| Recreate | One environment | Stop then replace | Slow | Downtime |
| Blue-green plus canary | Two complete | Small shift then full | Fast with evidence | Highest capacity/complexity |
```svg
```
**Selection and production application.** Use blue-green for high-value services with affordable duplicate capacity and compatible state; use canary for incremental exposure and rolling when capacity efficiency dominates. Inference services, APIs, web applications, control planes, registries and runtime upgrades use blue-green deployment. Success depends on router, environments, model artifacts, state, schemas, sessions, observability, capacity, automation and rollback. The useful optimization and reliability boundary is the complete user-facing system. Improving a model server, network, registry, deployment controller, or pipeline stage can move the bottleneck or weaken consistency, safety, recoverability, and cost elsewhere, so decisions are validated end to end. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.