numeracy analysis

**Numeracy Analysis** in NLP is the **systematic study and evaluation of how well language models understand, represent, and generate numerical information** — covering magnitude comparison, unit semantics, arithmetic, and number formatting, addressing the foundational weakness of statistical models that treat numbers as arbitrary token sequences rather than quantities on a linear scale. **What Is Numeracy in NLP?** Numeracy is distinct from mathematical problem-solving. It asks whether a model has an internal sense of number as a quantity: - **Magnitude Sense**: Does the model "know" that 1,000,000 is much larger than 100? - **Plausibility**: "A human weighs 70 kg" is plausible; "A human weighs 7,000 kg" is not — does the model recognize this? - **Unit Semantics**: Does the model understand that "70 mph" and "112 km/h" refer to the same speed? - **Arithmetic Grounding**: Can the model verify that 15% of 80 is 12, not just generate a plausible number? - **Ordinal Reasoning**: "Third fastest" implies a ranked ordering of speeds. **Why Tokenization Breaks Numeracy** Standard BPE tokenization fragments numbers in non-intuitive ways: - "1234" might tokenize as ["12", "34"] or ["1", "234"] depending on the vocabulary. - "10000" and "9999" — consecutive integers — may share no subword tokens and appear linguistically unrelated. - Magnitude is entirely implicit — the model must learn from context that "million" after a number means ×10⁶. This is fundamentally different from human number processing, where the digit positional system explicitly encodes magnitude. **Key Research Findings** - **Wallace et al. (2019) — "Do NLP Models Know Numbers?"**: Probed BERT embeddings for numeric knowledge. Found BERT has weak magnitude representations but can learn basic number comparison from fine-tuning. - **Thawani et al. (2021) — "Representing Numbers in NLP"**: Compared digit-by-digit encoding, scientific notation, numericalization (separate float embedding), and character models. No method dominates across all numeracy tasks. - **Berg-Kirkpatrick et al. — Scientific Numeracy**: Models hallucinate scientific numbers (atomic masses, physical constants) with alarming frequency, suggesting that number facts in pretraining are not reliably memorized. **Numeracy Failure Modes in Deployed LLMs** - **Unit Confusion**: "The population of China is approximately 1.4 billion" — models sometimes confuse million/billion/trillion in generation. - **Year Arithmetic**: "The policy was implemented 3 years after 2015" — models give inconsistent or wrong results. - **Percentage Errors**: "Double from 50% is 100%" — correct — but "increase 50% by 25%" is frequently miscalculated. - **Scale Blindness**: Generating "the building is 500 miles tall" without triggering implausibility detection. - **Context-Inconsistent Numbers**: Stating a statistic correctly in one paragraph and contradicting it in another. **Evaluation Tasks for Numeracy** - **Number Comparison**: "Which is larger: 3/7 or 0.45?" — tests rational number comprehension. - **Magnitude Estimation**: "A car weighs approximately ___ kg" — fill in a plausible range. - **Probing Classifiers**: Train a linear probe on model embeddings to predict whether a number is in a range — reveals implicit representational quality. - **Arithmetic Verification**: "Does 23 × 14 = 322?" — yes/no verification of calculation. - **NumGLUE (aggregated)**: Multi-task evaluation covering all numeracy dimensions. **Improvement Strategies** - **Digit-by-Digit Tokenization**: Represent "1234" as ["1", "2", "3", "4"] — preserves positional magnitude information. - **Scientific Notation Normalization**: Convert all numbers to `d.ddd × 10^n` before tokenization. - **Number-Span Embeddings**: Special embeddings that encode the parsed float value of a number token span. - **Tool Use**: Route numeric computation to a calculator or code interpreter — sidestep the representation problem entirely. - **Pretraining Data Engineering**: Include more mathematical and scientific text, tables, and spreadsheet data. Numeracy Analysis is **number sense for AI** — the critical research program ensuring that language models treat numbers as quantities with magnitude and units rather than arbitrary text sequences, addressing a foundational weakness that causes systematic hallucination in technical, financial, and scientific domains.