eval
**LLM Evaluation and Benchmarks**
**Why Evaluation Matters**
Rigorous evaluation ensures LLMs perform as expected on target tasks, helps compare models, and identifies areas for improvement.
**Standard Benchmarks**
**Knowledge and Reasoning**
| Benchmark | Description | Example Tasks |
|-----------|-------------|---------------|
| MMLU | Multitask, 57 subjects | History, math, law, medicine |
| HellaSwag | Commonsense reasoning | Sentence completion |
| ARC | Science questions | Elementary to college level |
| Winogrande | Pronoun resolution | Commonsense |
| TruthfulQA | Factual accuracy | Avoiding false claims |
**Code and Math**
| Benchmark | Description | Metric |
|-----------|-------------|--------|
| HumanEval | Python coding | Pass@k |
| MBPP | Basic Python | Pass@k |
| GSM8K | Grade school math | Accuracy |
| MATH | Competition math | Accuracy |
**Conversation and Instruction**
| Benchmark | Description |
|-----------|-------------|
| MT-Bench | Multi-turn conversation quality |
| AlpacaEval | Instruction following |
| Chatbot Arena | Human preference rankings |
**Evaluation Metrics**
**Automatic Metrics**
- **Perplexity**: Lower is better (language modeling quality)
- **Pass@k**: Probability of correct code in k attempts
- **BLEU/ROUGE**: Text similarity (limited usefulness for LLMs)
- **Exact Match**: For factual or extraction tasks
**Human Evaluation**
- **Preference rankings**: A vs B comparisons
- **Likert scales**: Quality ratings (1-5)
- **Task success rate**: Binary completion metrics
- **LLM-as-Judge**: Use GPT-4 or Claude to evaluate outputs
**Best Practices**
1. Use multiple benchmarks across capabilities
2. Include domain-specific evaluations for your use case
3. Combine automatic metrics with human judgment
4. Test for safety and edge cases, not just accuracy
5. Version evaluation sets and track performance over time