MATH is the competition-level mathematics benchmark of 12,500 problems drawn from AMC, AIME, and similar olympiad contests — designed to probe whether language models can perform creative, multi-step mathematical reasoning far beyond grade-school arithmetic, using problems that challenge even gifted human students.
What Is the MATH Dataset?
- Scale: 12,500 problems — 7,500 training, 5,000 test.
- Source: Problems from AMC 8, AMC 10, AMC 12, AIME, and HMMT competitions.
- Format: Free-form LaTeX input and solution, with a final boxed answer.
- Subjects: Algebra, Counting & Probability, Geometry, Intermediate Algebra, Number Theory, Prealgebra, Precalculus.
- Difficulty Levels: 1 (easiest) to 5 (hardest), where Level 5 problems require olympiad-level insight.
Why MATH Is Fundamentally Hard
Unlike arithmetic datasets (GSM8K, MAWPS) where the solution path is straightforward, MATH problems require:
- Insight Steps: "Notice that the expression is a perfect square" — non-obvious algebraic manipulations.
- Multiple Solution Strategies: Different approaches (substitution, induction, combinatorial argument) must be selected appropriately.
- Symbolic Precision: LaTeX output must be exactly correct — "$frac{3}{7}$" not "3/7".
- Long Solution Chains: Competition problems routinely require 10-15 logical steps, each building on the previous.
- Elegant Tricks: AMC/AIME problems often have "trick" solutions that brute-force arithmetic misses entirely.
Performance Timeline
| Model | Year | MATH Accuracy |
|-------|------|--------------|
| GPT-3 | 2020 | ~4.5% |
| Minerva 540B | 2022 | 33.6% |
| GPT-4 | 2023 | ~52% |
| GPT-4 with CoT | 2023 | ~67% |
| o1 (reasoning model) | 2024 | ~94.8% |
| Expert human (AMC/AIME competitor) | — | ~90-95% |
The jump from GPT-4 (~52%) to o1 (~95%) demonstrates that extended chain-of-thought reasoning — essentially letting the model "think longer" — is the key to breakthrough math performance.
Subject Breakdown (GPT-4 performance)
| Subject | Accuracy |
|---------|---------|
| Prealgebra | ~76% |
| Algebra | ~62% |
| Counting & Probability | ~50% |
| Number Theory | ~55% |
| Intermediate Algebra | ~42% |
| Precalculus | ~45% |
| Geometry | ~40% |
Geometry and advanced algebra remain the hardest subjects due to visual reasoning requirements and complex symbolic manipulation.
Why MATH Matters
- Genuine Reasoning Test: Math has unambiguous correct answers — no subjectivity, no annotation errors. A correct solution is definitively correct.
- Failure Mode Diagnosis: Early models scored near 0% on Level 5 problems despite 50%+ on Level 1, proving that scaling alone was insufficient — reasoning architecture mattered.
- Training Data for Reasoning: MATH's 7,500 training problems with full solution chains became a key fine-tuning resource for math-capable models (Minerva, WizardMath, DeepSeekMath).
- Verifiable Generation: Math is one of the few domains where AI output can be automatically verified with a symbolic solver — enabling reinforcement learning from correct solutions.
- Real-World Proxy: Mathematical reasoning ability correlates with performance on engineering, physics, and quantitative finance tasks.
Evaluation Techniques
- Majority Voting (Self-Consistency): Generate 40 solutions, take the most common answer — improves accuracy ~8-12%.
- Tool-Augmented: Allow code execution (Python sympy/numpy) — dramatically improves accuracy for algebraic manipulation.
- Process Reward Models (PRM): Train a verifier to score intermediate reasoning steps, not just final answers — enables beam search over solution paths.
Extensions and Variants
- MATH-500: Benchmark subset of 500 carefully selected problems for faster evaluation.
- MATH-Odyssey: Harder 2024 extension with post-2022 competition problems (avoiding contamination).
- OlympiadBench: Extends to International Mathematical Olympiad (IMO) level problems.
MATH is the mathematical olympiad for AI — a dataset that separates models that perform arithmetic from models that genuinely reason, with a clear, verifiable correctness criterion that enables rigorous measurement of progress toward human-level mathematical problem solving.