canary deployment

Canary deployment gradually rolls out a new model to small percentage of traffic, monitoring for problems before full release. **Process**: Deploy new model to 1-5% of traffic, monitor key metrics, if healthy increase to 10%, 25%, 50%, 100%. Rollback if issues. **Why canary**: Limits blast radius of problems. Real traffic validation. Quick rollback path. Builds confidence incrementally. **Metrics to monitor**: Latency, error rates, business metrics, user feedback. Compare canary to control population. **Automation**: Progressive delivery tools (Flagger, Argo Rollouts) automate percentage increases based on metrics. **Rollback triggers**: Error rate spike, latency increase, business metric degradation, manual halt. Automatic rollback possible. **Traffic routing**: Load balancer routes percentage to new model deployment. Consistent routing (same user always sees same version) often preferred. **Duration per stage**: Hours to days depending on traffic volume and confidence requirements. **Comparison to A/B testing**: Canary is for safe rollout, A/B is for choosing between versions. Can combine: canary new model, then A/B test features. **Best practice**: Start small, automate monitoring, have instant rollback ready.