gsm8k
**GSM8K** is a benchmark dataset of 8,500 linguistically diverse grade school math word problems designed to evaluate mathematical reasoning capabilities in language models.
## What Is GSM8K?
- **Size**: 7,500 training + 1,000 test problems
- **Difficulty**: Grade school level (ages 6-12)
- **Format**: Word problems requiring 2-8 step solutions
- **Metric**: Exact match accuracy on final numerical answer
## Why GSM8K Matters
Math reasoning is a key capability for AI assistants. GSM8K tests multi-step deductive reasoning that requires understanding language and applying arithmetic.
```
GSM8K Example Problem:
"Janet has 3 apples. She buys 2 more apples, then
gives half of all her apples to her friend. How
many apples does Janet have?"
Solution steps:
1. Start: 3 apples
2. Buy more: 3 + 2 = 5 apples
3. Give half: 5 / 2 = 2.5 → 2 apples (integer)
Answer: 2
```
**Model Performance (2024)**:
| Model | GSM8K Accuracy |
|-------|----------------|
| GPT-4 | ~92% |
| Claude 3 | ~88% |
| Llama 2 70B | ~57% |
| Chain-of-thought prompting | +10-20% improvement |