mathqa
**MathQA** is the **large-scale math word problem dataset annotated with executable operation programs** — bridging the gap between end-to-end answer prediction and interpretable program synthesis by requiring models to produce a structured formula tree that explicitly encodes the mathematical operations needed to solve each problem.
**What Is MathQA?**
- **Scale**: ~37,200 problems from AQuA-RAT, re-annotated with operation programs.
- **Format**: Multiple-choice question + natural language rationale + structured operation program.
- **Operation Language**: A domain-specific functional language: `divide(n1, n2)`, `multiply(n1, n2)`, `add(n1, subtract(n2, n3))` — composable arithmetic operations over extracted numbers.
- **Subjects**: Algebra, Arithmetic, Probability, Geometry, Physics, and General word problems.
- **Goal**: Map natural language problem text to an executable program that produces the correct answer.
**The Three-Part Annotation**
Each MathQA example contains:
1. **Problem Text**: "A train travels from city A to city B at 60 mph. The return trip is at 40 mph. What is the average speed for the entire trip?"
2. **Rationale (Natural Language)**: "Average speed = total distance / total time. Let d be the one-way distance. Time AB = d/60, time BA = d/40, total time = d/60 + d/40 = 5d/120. Average = 2d / (5d/120) = 48 mph."
3. **Operation Program**: `divide(multiply(2, 60), add(divide(60, 40), divide(40, 60)))` (simplified symbolic form)
**Why Operation Programs Matter**
Standard seq2seq math solvers (directly predicting the answer number) have three critical weaknesses:
- **Unverifiable**: A correct answer could come from wrong reasoning — no way to audit intermediate steps.
- **Non-compositional**: Cannot generalize to problems requiring a new combination of operations.
- **Brittle**: Small perturbations cause catastrophic failures because there's no structured representation to fall back on.
Operation programs address all three:
- **Auditable**: Every step of the computation is explicit and inspectable.
- **Compositional**: New problems can be solved by recombining known operations.
- **Executable**: The program can be run against a symbolic interpreter to verify correctness independently of the neural model.
**Why MathQA Matters**
- **Toward Neural Program Synthesis**: MathQA positioned math reasoning as a program synthesis problem, connecting NLP to the formal methods community.
- **Intermediate Representation**: Inspired later work on tool-augmented LLMs (code generation for math), where models write Python code rather than predict answers.
- **Few-Shot Curriculum**: The annotated rationales became a template for Chain-of-Thought fine-tuning.
- **Baseline Difficulty**: Even with structured program targets, seq2seq models achieve only ~70-75% accuracy — substantial error in a domain where the answer is always verifiable.
- **Dataset Noise Warning**: MathQA has known annotation inconsistencies — some operation programs do not match the natural language rationale. Researchers should use with caution and cross-reference.
**Performance Benchmarks**
| Approach | Accuracy |
|---------|---------|
| Human expert | ~95%+ |
| Seq2seq baseline | ~61% |
| BERT + program synthesis | ~73% |
| GPT-4 (direct answer) | ~85% |
| GPT-4 + code execution | ~92% |
**Connection to Downstream Work**
MathQA directly influenced:
- **PoT (Program-of-Thought)**: Generate Python code for math problems, execute for the answer.
- **PAL (Program-Aided Language Models)**: Use LLMs as code generators, Python interpreter as the solver.
- **Tool-Use Agents**: LLMs calling external calculators (Wolfram Alpha, sympy) for reliable numeric computation.
MathQA is **showing your mathematical work in executable form** — requiring the model to produce not just the answer but the precise sequence of operations that derives it, making math reasoning transparent, auditable, and composable.