DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring numerical reasoning operations like addition, counting, and sorting over text passages.
What Is DROP?
- Size: 96,000+ question-answer pairs
- Source: Wikipedia paragraphs (sports, history)
- Challenge: Requires arithmetic, not just text extraction
- Operations: Count, add, subtract, compare, sort
Why DROP Matters
Most QA benchmarks test text extraction. DROP tests whether models truly understand quantities and can perform discrete reasoning.
DROP Example:
Passage: "The Lions scored 14 points in the first
quarter, 7 in the second, and 21 in the third."
Question: "How many total points did the Lions
score in the first two quarters?"
Reasoning: 14 + 7 = 21
Traditional QA: Extract "14" or "7"
DROP: Compute 14 + 7 = 21 (not directly in text)
Model Performance (2024):
| Model | DROP F1 |
|---|---|
| GPT-4 | ~88% |
| Human | ~96% |
| BERT (original) | ~31% |
| NumNet+ | ~83% |
Key: Models need both reading comprehension AND numerical reasoning.
drop benchmarknumerical reasoningreading comprehension
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.