leaderboard
**LLM Leaderboards and Rankings**
**Major Leaderboards**
**Chatbot Arena (LMSYS)**
Human preference-based ranking using Elo scores:
- Users chat with two anonymous models
- Choose which response is better
- Elo rating updated based on votes
```
Leaderboard (example scores):
1. GPT-4o: 1290
2. Claude 3.5 Sonnet: 1271
3. Gemini 1.5 Pro: 1260
4. Llama 3.1 405B: 1250
...
```
**Open LLM Leaderboard (HuggingFace)**
Automated benchmarks for open models:
- MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K
**HELM (Stanford)**
Holistic evaluation with many metrics:
- Accuracy, calibration, robustness, fairness, efficiency
**Elo Rating System**
```python
def update_elo(winner_elo, loser_elo, k=32):
expected_winner = 1 / (1 + 10 ** ((loser_elo - winner_elo) / 400))
expected_loser = 1 - expected_winner
new_winner_elo = winner_elo + k * (1 - expected_winner)
new_loser_elo = loser_elo + k * (0 - expected_loser)
return new_winner_elo, new_loser_elo
```
**Interpreting Leaderboards**
| Elo Difference | Win Probability |
|----------------|-----------------|
| 0 | 50% |
| 100 | 64% |
| 200 | 76% |
| 400 | 91% |
**Leaderboard Limitations**
| Issue | Mitigation |
|-------|------------|
| Selection bias | Random sampling |
| Prompt diversity | Topic stratification |
| Position bias | Randomize A/B order |
| Length bias | Evaluate conciseness |
| Time | Ratings change over time |
**Domain-Specific Leaderboards**
| Domain | Leaderboard |
|--------|-------------|
| Coding | SWE-bench, LiveCodeBench |
| Math | MATH leaderboard |
| Safety | HarmBench |
| RAG | MTEB embeddings |
| Agents | AgentBench |
**Best Practices**
- Dont rely on single leaderboard
- Consider use case fit
- Check benchmark methodology
- Evaluate on your own data
- Monitor for gaming/overfitting