LLM Leaderboards and Rankings
Major Leaderboards
Chatbot Arena (LMSYS) Human preference-based ranking using Elo scores:
- Users chat with two anonymous models
- Choose which response is better
- Elo rating updated based on votes
Leaderboard (example scores):
1. GPT-4o: 1290
2. Claude 3.5 Sonnet: 1271
3. Gemini 1.5 Pro: 1260
4. Llama 3.1 405B: 1250
...
Open LLM Leaderboard (HuggingFace) Automated benchmarks for open models:
- MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K
HELM (Stanford) Holistic evaluation with many metrics:
- Accuracy, calibration, robustness, fairness, efficiency
Elo Rating System
def update_elo(winner_elo, loser_elo, k=32):
expected_winner = 1 / (1 + 10 ** ((loser_elo - winner_elo) / 400))
expected_loser = 1 - expected_winner
new_winner_elo = winner_elo + k * (1 - expected_winner)
new_loser_elo = loser_elo + k * (0 - expected_loser)
return new_winner_elo, new_loser_elo
Interpreting Leaderboards
| Elo Difference | Win Probability |
|---|---|
| 0 | 50% |
| 100 | 64% |
| 200 | 76% |
| 400 | 91% |
Leaderboard Limitations
| Issue | Mitigation |
|---|---|
| Selection bias | Random sampling |
| Prompt diversity | Topic stratification |
| Position bias | Randomize A/B order |
| Length bias | Evaluate conciseness |
| Time | Ratings change over time |
Domain-Specific Leaderboards
| Domain | Leaderboard |
|---|---|
| Coding | SWE-bench, LiveCodeBench |
| Math | MATH leaderboard |
| Safety | HarmBench |
| RAG | MTEB embeddings |
| Agents | AgentBench |
Best Practices
- Dont rely on single leaderboard
- Consider use case fit
- Check benchmark methodology
- Evaluate on your own data
- Monitor for gaming/overfitting
leaderboardarenaelo
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.