a/b testing for models
**A/B testing is a controlled randomized experiment that assigns eligible units to variants and estimates their causal effect on predefined outcomes.** It lets AI teams compare model, ranking, prompt, UX or policy versions on real traffic while separating treatment effect from time, population and operational noise. Variant A is commonly the control and B the candidate, but names do not establish validity; randomization unit, exposure, sample size, analysis and guardrails determine whether the conclusion is credible. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Predefine hypothesis, eligible population, unit, allocation, exposure, primary metric, guardrails, minimum detectable effect, power, duration, exclusions, stopping, multiple comparisons and decision rule.
**Architecture, control plane, and operating behavior.** An assignment service hashes or randomizes users, accounts or sessions into sticky cohorts; routing exposes variants; event instrumentation records assignment and outcomes; a metric pipeline joins data; an analysis service estimates effect and uncertainty; governance records the decision. Run an A/A instrumentation check, calculate sample size, launch with safety canary if needed, monitor invariant and guardrail metrics, avoid peeking or use valid sequential methods, complete the planned window, analyze intention-to-treat and segments, and decide with practical significance. A/B and multivariate tests use fixed random cohorts, multi-armed bandits adapt allocation toward reward, interleaving compares rankers within one result stream, switchback alternates time/location for marketplace interference, and shadow tests no user-visible treatment. The operational stack spans clients and producers, APIs or ingestion, queues and schedulers, stateless and stateful compute, accelerators, memory and storage, network fabrics, identity and policy, artifact registries, observability, automation, and human operations. Control-plane decisions and data-plane work are separated so overload or compromise in one does not silently corrupt the other. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone.
**Implementation, infrastructure, and failure modes.** Use deterministic assignment and exposure logs, prevent cross-device contamination where possible, deduplicate units, track sample-ratio mismatch, use CUPED or stratification carefully, cluster standard errors when assignment is grouped, and preserve experiment/config versions. Two models may double HBM residency, fragment batches and change latency or cost. Performance differences can mediate user outcomes, so infrastructure must be balanced and measured, not assumed equivalent. Selection bias, novelty and carryover, interference, missing events, sample-ratio mismatch, metric gaming, repeated peeking, underpowered segments, seasonality, multiple testing and deploying operationally unhealthy B create false results. Implementation favors immutable artifacts, declarative configuration, typed schemas, idempotent operations, bounded retries with jitter, deadlines, backpressure, health and readiness probes, least privilege, encrypted transport and storage, progressive rollout, reproducible environments, and complete telemetry. Automation has dry-run, approval, audit, and rollback paths. AI infrastructure joins CPUs, GPUs or NPUs, HBM, host memory, NICs and DPUs, PCIe and scale-up links, leaf-spine networks, local and shared storage, power delivery, and cooling. Topology, NUMA locality, bandwidth, failure domains, thermal headroom, and accelerator memory determine delivered behavior and must be visible to schedulers. Common failures include retry storms, queue collapse, stale health signals, split brain, partial writes, incompatible schemas, silent data corruption, time skew, dependency amplification, capacity fragmentation, noisy neighbors, credential leakage, unbounded state, monitoring blind spots, and recovery procedures that exist only on paper. A healthy component does not prove a healthy user journey.
**Verification, security, and lifecycle controls.** Test assignment uniformity/stickiness, event completeness, A/A null behavior, metric SQL against fixtures, exposure timing, bot/filter policy, power simulation, sequential procedure, rollback and reproduction from immutable data. Treatment effect, confidence or credible interval, p-value where appropriate, power, sample size, conversion/engagement/quality, latency, errors, safety, heterogeneity, sample-ratio mismatch and cost matter. Experiments require privacy, consent or lawful basis, minimization, fairness and harm review, exclusion of vulnerable cases, stopping authority, documentation, recourse and audit. Statistical significance does not override safety. Verification combines unit, contract and property tests, schema compatibility, load and soak tests, chaos and fault injection, security review, backup restoration, failover and rollback drills, dependency degradation, regional evacuation where applicable, data reconciliation, shadow traffic, canaries, and end-to-end synthetic checks. Tests run against production-like scale and permissions. Source, data, configuration, environment, model, registry metadata, infrastructure definition, dependency, image, driver, firmware, deployment, experiment, approval, incident, and rollback artifacts remain linked. Continuous controls detect drift, expired credentials, unowned resources, stale backups, regressions, policy exceptions, and unsupported versions. Owners define access, segregation of duties, data classification, residency, retention and deletion, vendor and supply-chain review, incident severity, communications, audit evidence, RTO/RPO or SLO exceptions, cost attribution, and change authority. Sensitive model and experiment artifacts receive the same integrity and confidentiality controls as source and production data.
| Method | Allocation | Primary goal | Strength | Main limitation |
|---|---|---|---|---|
| A/B test | Fixed randomized cohorts | Causal average effect | Clear inference | Needs sample/time |
| Multi-armed bandit | Adaptive reward allocation | Exploit while learning | Reduces opportunity cost | Biased/adaptive inference complexity |
| Interleaving | Mixed ranked results | Compare rankers sensitively | High power for search | Specialized outcome assumptions |
| Switchback | Alternating time/region | Handle marketplace interference | Cluster-level treatment | Time confounding/analysis |
| Shadow test | Copied traffic, no exposure | Operational validation | No user harm | Cannot measure user outcome |
| Canary release | Small progressive traffic | Limit release risk | Blast-radius control | Not automatically causal |
```svg
```
**Selection and production application.** Use A/B for stable causal comparison, canary first for release safety, bandits for ongoing reward allocation with understood inference tradeoffs, interleaving for sensitive ranking comparisons and switchbacks under network interference. Model versions, recommenders, search rankers, prompts, UI, pricing under proper governance, latency optimizations and agent policies use controlled experiments. Experiment validity spans assignment, routing, model deployment, events, data pipeline, metric definitions, statistics, guardrails and organizational decisions. The useful optimization and reliability boundary is the complete user-facing system. Improving a model server, network, registry, deployment controller, or pipeline stage can move the bottleneck or weaken consistency, safety, recoverability, and cost elsewhere, so decisions are validated end to end. A production definition states the service or pipeline boundary, tenants, workload and data classes, dependency graph, consistency and durability expectations, capacity envelope, latency and availability objectives, failure model, trust zones, deployment units, ownership, and evidence required for release. Architecture diagrams and service-level indicators must refer to the same boundary. Evaluation combines correctness and model quality with throughput, p50/p95/p99 latency, queue depth, saturation, availability, error and retry rates, freshness, data loss, recovery time, recovery point, capacity, utilization, memory, network, energy, cost, and operator toil. Service-level objectives use user-visible good events, explicit windows, and error budgets rather than infrastructure uptime alone. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.