numglue
**NumGLUE** is the **multi-task benchmark specifically targeting the numerical reasoning capabilities of NLP models** — aggregating 8 distinct datasets that require quantitative understanding embedded in natural language, exposing the systematic weakness of pre-BERT and early transformer models in treating numbers as meaningful quantities rather than arbitrary tokens.
**What Is NumGLUE?**
- **Scale**: ~101,000 examples across 8 tasks.
- **Format**: Multi-task evaluation — each task tests a different facet of numerical reasoning.
- **Motivation**: Standard NLU benchmarks (GLUE, SuperGLUE) contain minimal numerical content. NumGLUE fills this gap by explicitly requiring arithmetic, comparison, and quantitative inference.
**The 8 NumGLUE Tasks**
**Task 1 — Arithmetic QA (MathQA origins)**:
- Fill-in-the-blank math word problems.
- "If a car travels 60 mph for 2.5 hours, the distance traveled is ___ miles."
**Task 2 — Fill-in-the-Blank NLI**:
- Given a context with numbers, fill in a missing quantity that makes an entailment valid.
**Task 3 — Numerical QA (DROP-style)**:
- Discrete operations over reading comprehension passages: add, subtract, sort, count.
- "How many more points did Team A score than Team B?" over sports reports.
**Task 4 — Comparison (greater/less/equal)**:
- "A cheetah runs at 70 mph. A human runs at 10 mph. The cheetah runs ___ times faster."
**Task 5 — Listing / Sorting**:
- Sort a set of quantities in ascending or descending order from a paragraph.
**Task 6 — Number Conversion / Format**:
- Recognize equivalent representations (fractions, decimals, percentages).
**Task 7 — Unit Conversion**:
- "Convert 3.5 miles to kilometers." Requires world knowledge of conversion factors.
**Task 8 — Quantitative NLI**:
- "Context states 5 million people. Does it entail that more than 3 million are affected?" Binary yes/no.
**Why NumGLUE Matters**
- **Tokenization Blindness**: Standard BPE tokenizers split numbers into sub-word pieces ("1995" → "19" + "95") losing magnitude information. NumGLUE highlighted this as a systematic failure mode.
- **Embedding Space Numbers**: Research (Wallace et al., 2019) showed that BERT representations lack a coherent linear number line — numbers close in value are not close in embedding space. NumGLUE quantified the performance consequence.
- **Cross-Task Transfer**: A model that handles arithmetic well should also handle comparison well (they require the same underlying magnitude understanding). NumGLUE tests whether this transfer actually occurs.
- **Real-World Ubiquity**: Numbers appear everywhere — financial reports, scientific papers, news articles, contracts. A model without numerical grounding fails on all of these.
- **Hallucination Root Cause**: LLMs that generate plausible-sounding but numerically wrong facts (dates, statistics, measurements) often fail because of the exact weaknesses NumGLUE measures.
**Performance Results**
| Model | NumGLUE Average |
|-------|----------------|
| T5-base | ~55% |
| GPT-3 175B | ~62% |
| UnifiedQA (T5 large) | ~67% |
| NumBERT (number-aware BERT) | ~71% |
| GPT-4 | ~85%+ |
**Improvements from Number-Aware Architecture**
Specialized models (NumBERT, GenBERT) that modify tokenization for numbers (digit-by-digit encoding, numericalized representations, injection of number magnitude embeddings) consistently outperform standard transformer baselines by 8-15 points.
**Connection to DROP and TATQA**
NumGLUE overlaps conceptually with:
- **DROP (Discrete Reasoning Over Paragraphs)**: Reading comprehension with numerical operations.
- **TATQA**: Table and text QA with financial arithmetic.
- **FinQA**: Financial report numerical reasoning.
All require numerical grounding; NumGLUE is distinctive in explicitly categorizing the required operation type across 8 distinct dimensions.
NumGLUE is **literacy plus numeracy combined** — testing the critical intersection where language understanding meets quantitative reasoning, ensuring AI models can handle the numerical fabric of real-world text rather than treating every number as an arbitrary symbol.