mawps

**MAWPS (Math Word Problem Repository)** is the **unified testbed for evaluating arithmetic word problem solvers** — aggregating multiple elementary math datasets (AddSub, MultiArith, SingleOp, SingleEq) into a standardized repository that enabled systematic comparison of semantic parsing, neural seq2seq, and symbolic AI approaches to math reasoning. **What Is MAWPS?** - **Scale**: ~3,320 elementary school math word problems across multiple sub-datasets. - **Operations**: Single and multi-step arithmetic — addition, subtraction, multiplication, division. - **Difficulty**: Grade school level (ages 6-12); no algebraic variables, no competition-level insight required. - **Format**: Natural language problem statement → numeric answer. - **Sub-datasets Included**: - **AddSub**: Single-step addition and subtraction (395 problems). - **MultiArith**: Multi-step problems requiring multiple operations (600 problems). - **SingleOp**: One-operation problems from diverse sources (562 problems). - **SingleEq**: Single-equation problems with one unknown (508 problems). **The Semantic Parsing Tradition** MAWPS was created in an era when the dominant approach to math word problems was semantic parsing — converting text into formal representations: - **Template Mapping**: "John has X apples and gives Y to Mary. How many does John have?" → `X - Y = ?` - **Equation Trees**: Represent the solution as a tree of arithmetic operations. - **Parse + Execute**: Translate text to equation, then evaluate the equation. The repository unified these approaches by providing standardized train/test splits across all sub-datasets, enabling direct comparison. **Why MAWPS Was Strategically Important** - **Baseline Establishment**: Before MAWPS, each paper used different datasets with incompatible splits. MAWPS created a common ground for comparison. - **Saturation Demonstration**: By 2020-2022, neural models (fine-tuned BERT, GPT-3) achieved ~95%+ accuracy on MAWPS — demonstrating that elementary arithmetic is essentially "solved" for LLMs. - **Stepping Stone**: MAWPS→GSM8K→MATH represents a progression — MAWPS confirmed arithmetic capability, motivating harder benchmarks. - **Neural vs. Symbolic**: MAWPS was a key arena for comparing end-to-end neural approaches (seq2seq) against symbolic semantic parsers — neural won by a significant margin for simple problems. **Performance by Model Generation** | Model | MAWPS Accuracy | |-------|---------------| | SVM expression classifier (2015) | ~73% | | Seq2Tree LSTM (2016) | ~88% | | BERT fine-tuned (2020) | ~93% | | GPT-3 few-shot (2022) | ~94% | | GPT-4 (2023) | ~98%+ | **MAWPS in the Current Context** As a near-solved benchmark, MAWPS serves specific purposes: - **Regression Testing**: Verify that new models do not lose basic arithmetic capability. - **Cross-lingual Transfer**: Translate MAWPS into other languages to measure arithmetic transfer without algebraic complexity. - **Few-Shot Lower Bound**: Measure how few examples a model needs to correctly solve grade-school arithmetic — tests sample efficiency. - **Error Analysis**: The remaining ~2-5% errors reveal systematic failure modes (negative numbers, implicit unit conversions, ambiguous plurals). **Common Failure Patterns** - **Implicit Units**: "John bought 3 dozen eggs." Models sometimes fail to multiply by 12. - **Comparison to Reference**: "Mary has 5 more apples than John, who has 8." Requires tracking two quantities. - **Multi-step Chaining**: 4+ operation problems in MultiArith expose breakdown in intermediate result tracking. **Relationship to Other Benchmarks** | Benchmark | Difficulty | Focus | |-----------|-----------|-------| | MAWPS | Elementary | Arithmetic | | GSM8K | Middle school | Multi-step arithmetic | | SVAMP | Elementary + adversarial | Robustness | | MATH | Competition level | Creative reasoning | | AQuA-RAT | GRE/GMAT | Algebraic reasoning | MAWPS is **the elementary math class benchmark** — historically essential for establishing arithmetic NLP baselines, now primarily serving as a sanity check confirming that modern LLMs have thoroughly mastered grade-school arithmetic word problems.