A/B testing compares two model versions by randomly showing different outputs to users and measuring outcomes. Process: Split users randomly into groups A and B, show each group outputs from different models, measure business/quality metrics, determine statistical significance. Metrics to track: User satisfaction, task completion, engagement, retention, conversion, explicit feedback. Statistical rigor: Calculate sample size for power, run until significant, account for multiple comparisons, use appropriate statistical tests. Online vs offline: Offline testing uses static datasets, online testing uses real users in production. Online captures real-world behavior. Challenges: User experience during test, long-term effects, novelty effects, segment differences. Interleaving: Show both model outputs to same user, let them choose. More efficient but not always applicable. Multi-armed bandits: Adaptive allocation that exploits better performing variant while still exploring. When to use: Before major model updates, comparing architectures, parameter changes, prompt variations. Best practices: One variable at a time, sufficient sample size, document and review results. Industry standard for production decisions.