big-bench

BIG-Bench (Beyond the Imitation Game Benchmark) is a collaborative benchmark containing over 200 diverse and challenging tasks designed to probe language model capabilities and limitations across a vast range of cognitive domains, from linguistics and mathematics to social reasoning and scientific understanding. Created through a community effort involving over 450 authors from 132 institutions, BIG-Bench was introduced in 2022 as an attempt to systematically discover what large language models can and cannot do across tasks chosen to be beyond the capabilities of current models. Tasks span categories including: traditional NLP (translation, summarization, question answering), mathematics and logic (arithmetic, logical deduction, cryptography), scientific reasoning (cause and effect, physical intuition, scientific literacy), social reasoning (social intelligence, sarcasm detection, moral judgment), world knowledge (sports, history, geography, medicine), creativity (analogies, humor, creative writing), reading comprehension (multi-hop reasoning, implicit reasoning), and meta-cognitive tasks (calibration, self-awareness, task identification). BIG-Bench Hard (BBH) is a curated subset of 23 tasks that were found to be particularly challenging for language models — tasks where models showed flat or below-human performance even at the largest scales. Key findings from BIG-Bench include: emergent capabilities (some tasks show near-zero performance for small models and then sudden improvement at specific scale thresholds), chain-of-thought prompting dramatically improves performance on reasoning-heavy tasks, and some tasks remain resistant to scaling (suggesting they require capabilities that current architectures lack). The benchmark uses both exact match and model-graded evaluation depending on the task. BIG-Bench has been instrumental in understanding emergent behaviors in language models — demonstrating that certain capabilities appear unpredictably at specific scales — and in identifying persistent weaknesses that guide research directions for improving reasoning, calibration, and multi-step problem-solving.