closed-book qa
**Closed-Book QA** is a question-answering paradigm where a language model must answer factual questions using only the knowledge stored in its parameters during pre-training, without access to any external documents, knowledge bases, or retrieval mechanisms at inference time. The model's parameters serve as an implicit knowledge base, and performance depends entirely on how much factual knowledge was absorbed and retained during pre-training.
**Why Closed-Book QA Matters in AI/ML:**
Closed-Book QA serves as a **critical benchmark for measuring the factual knowledge capacity** of language models, revealing how effectively large-scale pre-training encodes world knowledge in model parameters and highlighting the limitations of parametric-only knowledge storage.
• **Parametric knowledge storage** — Large language models (GPT, T5, PaLM) store factual knowledge implicitly in their weight matrices during pre-training on massive text corpora; closed-book QA tests how accurately this knowledge can be recalled through natural language generation
• **Scale-dependent performance** — Closed-book QA performance scales strongly with model size: T5-11B achieves significantly higher accuracy than T5-small on TriviaQA and Natural Questions, demonstrating that larger parameter spaces store more retrievable factual knowledge
• **Knowledge boundaries** — Closed-book QA exposes systematic knowledge gaps: models struggle with rare entities, recent events (post-training cutoff), numerical facts, and multi-step factual reasoning, revealing where parametric knowledge storage fails
• **Comparison baseline** — Closed-book performance establishes the parametric knowledge baseline against which retrieval-augmented (open-book) approaches are measured, quantifying the value added by external knowledge access
• **Hallucination risk** — Without retrieval grounding, closed-book models may generate plausible but incorrect answers (hallucinations), making this paradigm particularly prone to confident factual errors that are difficult to detect
| Model | Natural Questions (EM) | TriviaQA (EM) | Paradigm |
|-------|----------------------|---------------|----------|
| T5-Base (220M) | 25.2% | 23.4% | Closed-book |
| T5-Large (770M) | 29.8% | 28.5% | Closed-book |
| T5-11B | 34.5% | 42.3% | Closed-book |
| GPT-3 (175B) | 29.9% | 71.2% | Closed-book |
| DPR + Reader | 41.5% | 57.9% | Open-book |
| RAG | 44.5% | 56.1% | Open-book (retrieval) |
**Closed-book QA is the fundamental benchmark for evaluating how effectively language models encode and retrieve factual knowledge purely from parameters, establishing baseline performance that motivates retrieval-augmented approaches and revealing the inherent limitations of storing world knowledge entirely in neural network weights.**