drop benchmark
**DROP (Discrete Reasoning Over Paragraphs)** is a reading comprehension benchmark requiring numerical reasoning operations like addition, counting, and sorting over text passages.
## What Is DROP?
- **Size**: 96,000+ question-answer pairs
- **Source**: Wikipedia paragraphs (sports, history)
- **Challenge**: Requires arithmetic, not just text extraction
- **Operations**: Count, add, subtract, compare, sort
## Why DROP Matters
Most QA benchmarks test text extraction. DROP tests whether models truly understand quantities and can perform discrete reasoning.
```
DROP Example:
Passage: "The Lions scored 14 points in the first
quarter, 7 in the second, and 21 in the third."
Question: "How many total points did the Lions
score in the first two quarters?"
Reasoning: 14 + 7 = 21
Traditional QA: Extract "14" or "7"
DROP: Compute 14 + 7 = 21 (not directly in text)
```
**Model Performance (2024)**:
| Model | DROP F1 |
|-------|---------|
| GPT-4 | ~88% |
| Human | ~96% |
| BERT (original) | ~31% |
| NumNet+ | ~83% |
Key: Models need both reading comprehension AND numerical reasoning.