aqua-rat
**AQuA-RAT (Algebra Question Answering with Rationales)** is the **100,000-question algebra dataset where every problem comes with a human-written natural language rationale explaining the solution step-by-step** — one of the foundational datasets that demonstrated how explicit reasoning steps improve both model training and interpretability, directly inspiring the Chain-of-Thought prompting paradigm.
**What Is AQuA-RAT?**
- **Scale**: ~100,000 algebra and arithmetic problems (large for its era).
- **Format**: Multiple-choice (5 options: A/B/C/D/E) + free-form natural language rationale.
- **Source**: Problems crowdsourced via Amazon Mechanical Turk and adapted from GRE/GMAT preparation materials.
- **Coverage**: Ratio and proportion, percent, average, speed/distance/time, profit and loss, linear equations, simple probability.
- **Rationale Format**: "First, let x = the original price. Then 0.8x = 40, so x = 50. The answer is C."
**The Rationale Innovation**
Before AQuA-RAT, math datasets provided only (problem, answer) pairs. AQuA-RAT added the critical third element: the reasoning chain. This enables:
- **Process Supervision**: Train models on correct intermediate steps, not just final answers.
- **Error Attribution**: When a model is wrong, examine the rationale to find where reasoning broke down.
- **CoT Template Generation**: AQuA-RAT rationales served as templates for manually crafting Chain-of-Thought few-shot examples in Wei et al. (2022), the seminal CoT paper.
- **Student Modeling**: Educational AI can compare a student's reasoning chain to the gold rationale to identify misconceptions.
**Connection to Chain-of-Thought**
The 2022 paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" used AQuA-RAT as one of its five benchmark tasks. The key insight — that providing step-by-step reasoning examples in the prompt dramatically improved LLM performance on math problems — was demonstrated on AQuA-RAT alongside GSM8K, SVAMP, MAWPS, and MATH.
| Prompting Method | AQuA-RAT Accuracy (PaLM 540B) |
|-----------------|-------------------------------|
| Standard few-shot | 35.0% |
| Chain-of-Thought | 56.9% |
| Self-consistency (40 paths) | 73.2% |
**Why AQuA-RAT Matters**
- **Historical Significance**: One of the first large-scale datasets with natural language reasoning annotations for math — pioneered the idea that explanations improve AI math performance.
- **GRE/GMAT Difficulty**: Problems are at the standardized test level, requiring algebraic setup (not just arithmetic). This is harder than primary-school word problems (MAWPS) but accessible without competition-level insight (MATH).
- **Multi-Step Reasoning**: Most problems require 3-5 logical steps, making them ideal for CoT evaluation.
- **Curriculum Learning**: The rationale quality varies (crowdsourced annotation has noise), making AQuA-RAT useful for studying how model performance degrades with noisy supervision.
- **Broad Coverage**: GRE/GMAT topics are directly relevant to standardized test preparation AI and educational technology.
**Known Limitations**
- **Annotation Noise**: Some crowdsourced rationales contain arithmetic errors or unclear steps (~5-10% estimated noisy examples).
- **Limited Symbolic Diversity**: Compared to MATH (competition level), AQuA-RAT problems are formulaic — the same problem structures repeat with different numbers.
- **English Only**: No multilingual variants, limiting use in international educational AI research.
**Datasets It Inspired**
- **MathQA**: Re-annotated AQuA-RAT problems with structured operation programs.
- **GSM8K**: More carefully crowdsourced grade-school math with clean step-by-step rationales.
- **ORCA**: Used AQuA-RAT-style rationale generation at scale with LLM-generated explanations.
AQuA-RAT is **the algebra textbook that taught AI to show its work** — proving that natural language reasoning chains are not just interpretability aids but genuine performance boosters, laying the intellectual foundation for the Chain-of-Thought era of language model development.