gopher

Gopher is DeepMind's 280 billion parameter language model introduced in 2021, designed to study the relationship between model scale and performance across a comprehensive set of 152 evaluation tasks spanning language understanding, reading comprehension, mathematical reasoning, scientific knowledge, common sense, logical reasoning, and ethical reasoning. While primarily a research model, Gopher provided critical insights about the benefits and limitations of scaling language models. Gopher's architecture is a standard autoregressive transformer decoder trained on MassiveText — a diverse, high-quality dataset of 10.5 TB comprising web pages (filtered with quality classifiers), books, news articles, code (GitHub), and Wikipedia. DeepMind also trained smaller models at 44M, 117M, 417M, 1.4B, 7.1B, and 280B parameters to systematically study scaling behavior. Key findings from the Gopher paper included: scaling provides non-uniform benefits across tasks (knowledge-intensive tasks like fact retrieval and reading comprehension improved dramatically with scale, while mathematical reasoning and logical inference showed more modest gains — suggesting these require capabilities beyond pattern matching), larger models are more data-efficient (achieving given performance levels with fewer training examples), and even at 280B parameters, the model had significant limitations in multi-step logical reasoning, numerical computation, and tasks requiring grounded understanding. Gopher achieved state-of-the-art on approximately 100 of 152 evaluation tasks at its release, particularly excelling on knowledge-intensive benchmarks like MMLU. The model was later shown to be undertrained by the Chinchilla analysis — the same compute used for Gopher's 280B parameters could achieve better results with a 70B model trained on 4.7× more data. Gopher's comprehensive evaluation framework and honest analysis of scaling limitations significantly influenced the field's understanding of what scale can and cannot achieve in language modeling.