big-bench

**BIG-bench (Beyond the Imitation Game Benchmark)** is a **collaborative benchmark consisting of 200+ diverse tasks designed to probe the capabilities and limitations of large language models** — created by hundreds of researchers submitting "tasks where humans excel but LLMs fail". **Diversity** - **Tasks**: Emoji movie guessing, chess state tracking, irony detection, swahili translation, biology, physics. - **Hard**: Specifically designed to be "future-proof" — many tasks were near 0% performance for GPT-3. - **Lite**: BIG-bench Lite is a distinct subset of roughly 24 tasks used for cheaper evaluation. **Why It Matters** - **Broadness**: Moving away from just "GLUE" (NLU) to "Everything" (Reasoning, Humor, Coding). - **Emergence**: Used to study "Emergent Abilities" — skills that suddenly appear only at scale (10B+ params). - **Canary**: Uses a "canary string" to prevent the benchmark data from leaking into future training sets. **BIG-bench** is **the gauntlet** — a massive, community-driven suite of weird and hard checks to find the breaking points of Large Language Models.