abtest
**A/B Testing and LLM Rollouts**
**A/B Testing for LLMs**
Unlike traditional software, LLM outputs are non-deterministic and subjective. A/B testing helps you make data-driven decisions about prompt changes, model upgrades, and parameter tuning.
**Experiment Design**
**What to Test**
| Variable | Examples |
|----------|----------|
| Model | GPT-4 vs Claude-3 |
| Prompt | Short vs detailed system prompt |
| Parameters | Temperature 0.3 vs 0.7 |
| Architecture | Direct call vs RAG |
**Metrics to Compare**
1. **Task Metrics**: Accuracy, success rate
2. **User Metrics**: Thumbs up/down, NPS
3. **Performance**: Latency, cost
4. **Safety**: Guardrail violations
**Statistical Considerations**
- **Sample size**: Need enough users/requests for significance
- **Duration**: Run long enough to capture variance
- **Segmentation**: Consider user segments separately
- **Multiple hypothesis correction**: Adjust p-values for multiple metrics
**Canary Deployments**
**Rollout Strategy**
```
[New Model Version]
↓
[Deploy to 1%] → Monitor → [Issues?] → Rollback
↓
[Increase to 10%] → Monitor
↓
[Increase to 50%] → Monitor
↓
[Full rollout at 100%]
```
**Monitoring During Rollout**
| Stage | Traffic | Duration | Metrics to Watch |
|-------|---------|----------|------------------|
| Canary | 1% | 1 hour | Error rate, latency |
| Limited | 10% | 4 hours | User feedback |
| Broad | 50% | 1 day | Full metric suite |
| Full | 100% | Ongoing | Continuous monitoring |
**Feature Flags for LLMs**
```python
**Example using LaunchDarkly pattern**
if feature_flags.is_enabled("use_gpt4_turbo", user_id):
model = "gpt-4-turbo"
else:
model = "gpt-4"
```
**Online Evaluation**
**LLM-as-Judge**
Use a capable LLM to evaluate outputs:
```
Rate the following response on helpfulness (1-5):
Question: {question}
Response: {response}
Rating:
```
**Human Evaluation Sampling**
- Sample 1-5% of requests for human review
- Use rating scales (1-5) for consistency
- Track inter-annotator agreement
**Tools for Experimentation**
| Tool | Type | Features |
|------|------|----------|
| LaunchDarkly | Feature flags | Enterprise, targeting |
| Statsig | Experimentation | Statistics focus |
| Growthbook | Open source | Self-hostable |
| Eppo | AI-focused | LLM metrics built-in |