a/b testing for models

A/B testing for models compares multiple deployed versions to determine which performs better with real users. **Setup**: Split traffic randomly between versions A and B, measure business-relevant metrics, run until statistically significant. **Metrics to compare**: User engagement, conversion rate, task completion, satisfaction surveys, downstream business metrics. Not just model accuracy. **Statistical rigor**: Power analysis for sample size, significance testing (t-test, chi-square), confidence intervals, watch for multiple comparison issues. **Duration**: Run long enough for significance and to capture time patterns. Too short may miss weekly cycles. **Traffic split**: Often 50/50 for speed, but can use 90/10 for safety (test with minority). **Guardrail metrics**: Safety metrics that must not degrade (latency, errors, safety violations). Halt if violated. **Multi-armed bandits**: Adaptive approach that shifts traffic toward better-performing variant during experiment. **Segmentation**: Analyze results by user segments, may find variant works better for some users. **Infrastructure**: Feature flags, traffic routing, metric collection, experiment management platform. **Documentation**: Record hypothesis, results, decision, learnings.