a/b testing

A/B testing compares two model versions by randomly showing different outputs to users and measuring outcomes. **Process**: Split users randomly into groups A and B, show each group outputs from different models, measure business/quality metrics, determine statistical significance. **Metrics to track**: User satisfaction, task completion, engagement, retention, conversion, explicit feedback. **Statistical rigor**: Calculate sample size for power, run until significant, account for multiple comparisons, use appropriate statistical tests. **Online vs offline**: Offline testing uses static datasets, online testing uses real users in production. Online captures real-world behavior. **Challenges**: User experience during test, long-term effects, novelty effects, segment differences. **Interleaving**: Show both model outputs to same user, let them choose. More efficient but not always applicable. **Multi-armed bandits**: Adaptive allocation that exploits better performing variant while still exploring. **When to use**: Before major model updates, comparing architectures, parameter changes, prompt variations. **Best practices**: One variable at a time, sufficient sample size, document and review results. Industry standard for production decisions.