mmlu
**MMLU (Massive Multitask Language Understanding)** is the **benchmark of 57 academic and professional subjects — from elementary mathematics to medical licensing exams — that became the de facto standard for measuring LLM knowledge depth and breadth** — first exposing the massive gap between early language models and human expert performance, then tracking the rapid progress that brought AI to near-expert levels within three years.
**What Is MMLU?**
- **Scale**: 15,908 multiple-choice questions across 57 subjects.
- **Format**: 4-option multiple-choice (A/B/C/D) with a single correct answer.
- **Subjects**: Organized into four domains — STEM (math, physics, chemistry, biology, computer science), Humanities (history, philosophy, law), Social Sciences (economics, psychology, sociology), and Professional (medical licensing, legal bar, accounting).
- **Difficulty**: Ranges from high-school level (elementary mathematics) to professional certification level (USMLE, LSAT, CPA exams).
- **Human Baseline**: Non-expert humans score ~34.5% (essentially random for hard topics); expert humans score ~89.8%.
**The 57 Subjects**
**STEM**:
- Abstract Algebra, College Chemistry, College Mathematics, College Physics, Computer Security, Electrical Engineering, High School Biology, High School Chemistry, Machine Learning, Virology
**Humanities**:
- High School World History, International Law, Jurisprudence, Logical Fallacies, Moral Disputes, Philosophy, Prehistory, World Religions
**Social Sciences**:
- Econometrics, High School Government and Politics, Human Sexuality, Professional Psychology, Sociology
**Professional / Applied**:
- Clinical Knowledge, Medical Genetics, Anatomy, Professional Medicine, Professional Law, Professional Accounting, Nutrition, Management
**Why MMLU Became the Standard**
- **GPT-3 Failure (2020)**: When MMLU was released, GPT-3 (175B parameters) scored ~43% — barely above random chance on hard subjects. This galvanized the field.
- **Single Number Comparability**: MMLU provides one average accuracy across all 57 subjects — making it easy to compare models in papers and leaderboards.
- **Knowledge vs. Reasoning**: MMLU tests factual recall AND multi-step reasoning (medical diagnosis questions, legal analysis). This dual test exposes models that rely solely on pattern matching.
- **Broad Coverage**: No single training set can cover all 57 domains — MMLU tests genuine cross-domain knowledge transfer.
- **Progressive Bar**: GPT-4 (~86%+), Claude 3 Opus (~88%), Gemini Ultra (~90%) approaching but not exceeding average expert human performance.
**Performance Timeline**
| Model | Year | MMLU Score |
|-------|------|-----------|
| GPT-3 175B | 2020 | 43.9% |
| InstructGPT | 2022 | 52.0% |
| GPT-3.5 | 2022 | 70.0% |
| GPT-4 | 2023 | 86.4% |
| Claude 3 Opus | 2024 | 88.2% |
| Gemini Ultra | 2024 | 90.0% |
| Expert Human | — | ~89.8% |
**MMLU Variants and Extensions**
- **MMLU-Pro**: Harder version with 10 answer choices and more reasoning-heavy questions.
- **MMLU-Redux**: Cleaned version fixing annotation errors in the original (~450 questions re-evaluated).
- **Multilingual MMLU**: Translated versions testing cross-lingual knowledge transfer.
- **Domain-Specific**: Medical MMLU, Legal MMLU subsets for specialized evaluation.
**Limitations**
- **Knowledge Contamination**: MMLU questions appear in many pretraining corpora; models may have memorized answers rather than reasoning to them.
- **Answer Format Bias**: 4-choice format allows positional biases ("C is always correct" patterns in some models).
- **No Explanation Required**: Correct answer without reasoning path — models can be right for wrong reasons.
- **Static Knowledge**: Questions frozen at release date — medical and legal knowledge evolve, making some answers outdated.
**Evaluation Best Practices**
- **5-shot Prompting**: Standard evaluation uses 5 few-shot examples per subject to establish format.
- **Chain-of-Thought**: MMLU-CoT variants require step-by-step reasoning before selecting the answer.
- **Calibration**: Strong models should be well-calibrated — high confidence on questions they answer correctly.
MMLU is **the comprehensive IQ test for language models** — measuring not just what a model has memorized but whether it can integrate knowledge across 57 disciplines to correctly answer questions that require the depth of a medical professional, lawyer, or scientist.