GOAT (Good at Arithmetic Tasks) is a Llama-based language model fine-tuned specifically for arithmetic calculation, demonstrating that targeted synthetic data training can solve the fundamental tokenization problem that makes standard LLMs fail at basic math — achieving state-of-the-art performance on multi-digit addition, subtraction, multiplication, and division by training on carefully structured arithmetic examples that teach the model columnar computation strategies, even outperforming GPT-4 on certain large-number operations at time of release.

Why LLMs Fail at Arithmetic

• Tokenization Problem: Standard LLMs tokenize "12345" as subword chunks like "123" + "45" or "1" + "2345" — destroying the digit-level alignment needed for columnar arithmetic. The model literally cannot see individual digits in consistent positions.
• Pattern vs. Computation: LLMs learn statistical patterns, not algorithms. They memorize that "2+2=4" from training data but cannot generalize to "47293+81956" because that specific sum was never in training.
• Carry Propagation: Multi-digit addition requires carrying across columns — a sequential, algorithmic process that autoregressive generation handles poorly without explicit training.

The GOAT Solution

Key Innovation: GOAT's training data presents arithmetic problems with explicit intermediate steps — showing the model how to align digits, propagate carries, and verify results. This transforms arithmetic from pattern-matching into learned algorithmic execution.

Performance

Significance: GOAT proved that domain-specific fine-tuning on synthetic data can solve fundamental LLM limitations — the tokenization problem isn't inherent to the architecture but addressable through targeted training. This influenced subsequent math-specialized models (MAmmoTH, MetaMath, Llemma) and validated the approach of using synthetic datasets to teach LLMs algorithmic reasoning.

GOAT is a landmark demonstration that LLMs can learn genuine computation — proving that fine-tuning with structured arithmetic examples enables models to perform reliable multi-digit calculation that base models and even frontier systems struggle with, establishing synthetic data as the key to teaching algorithmic skills.