NarrativeQA (Long)

NarrativeQA (Long) is the full-document variant of the NarrativeQA benchmark — requiring models to read entire movie scripts or Gutenberg novels averaging 50,000-80,000 words to answer free-form questions, representing the frontier challenge of long-document comprehension where the answer may be embedded anywhere in a text far exceeding the context window of standard models.

What Is NarrativeQA?

- Origin: Kočiský et al. (2018) from DeepMind.
- Scale: 1,567 stories (783 books + 789 movie scripts) with 46,765 question-answer pairs.
- Format: Each story has ~30 questions; answers are free-form text (averaging ~4 words), not multiple-choice.
- Answer Source: Questions were written by human annotators who read only the plot summary — ensuring questions probe deep story understanding, not surface pattern matching.
- Two Evaluation Variants: Context = summary (~700 words) OR context = full text (~50,000-80,000 words).

Why the Long Version Is Hard

The "Long" setting — using the full book or script rather than a summary — exposes three fundamental challenges:

Challenge 1 — Context Window Overflow:
- Most transformer models cap at 4k-8k tokens (~3k-6k words). A 60,000-word novel = ~80,000 tokens.
- Solutions: RAG (retrieve relevant passages), sliding window attention, hierarchical summarization, or very long context models (Claude 100k, Gemini 1M).

Challenge 2 — Holistic Understanding:
- Some questions require synthesizing character development from chapter 1 and chapter 30: "How did [character] change throughout the story?"
- RAG retrieval of top-3 passages cannot answer these — the entire arc is needed.

Challenge 3 — Needle in a Haystack:
- Specific factual questions ("What was the name of the detective's partner's dog?") require finding a single sentence in 80,000 words.
- Retrieval can find this efficiently, but with ~5% retrieval failure rate, 5% of answers become impossible.

Performance Results

| Model | Setting | ROUGE-L | BLEU-1 | METEOR |
|-------|---------|---------|--------|--------|
| SeqToSeq baseline | Summary | 28.5 | 23.8 | 21.5 |
| BiDAF | Summary | 36.6 | 33.7 | 28.7 |
| GPT-3.5 | Full text (RAG) | 42.1 | 38.4 | 33.2 |
| GPT-4 | Full text (RAG) | 52.3 | 48.1 | 41.6 |
| Claude 2 100k | Full text (no retrieval) | 59.4 | 54.8 | 48.3 |
| Human | Summary | 67.0 | 62.9 | 55.8 |

Evaluation Metrics

NarrativeQA uses three complementary metrics because answers are free-form and often have multiple valid phrasings:
- BLEU: N-gram precision between generated answer and reference answers.
- ROUGE-L: Longest common subsequence recall.
- METEOR: Unigram recall with stemming and synonym matching.

Why NarrativeQA (Long) Matters

- Ultimate Long-Context Test: No benchmark better distinguishes models with 8k vs. 100k context windows than NarrativeQA long — the performance gap is stark and meaningful.
- Literary Understanding: Books contain subtle character psychology, narrative irony, and thematic arcs that require understanding the whole text — a genuine test of deep reading comprehension.
- Application Relevance: AI research assistants, legal discovery (reading full case files), and educational summarization all require NarrativeQA-style full-document comprehension.
- RAG Architecture Driver: NarrativeQA long motivated significant research into passage retrieval optimization, dense passage indexing, and hierarchical document representation.
- Context Utilization Research: NarrativeQA long is used to study "lost in the middle" — the finding that models best use information at the beginning and end of context, missing information in the middle of long documents.

Famous "Needle in a Haystack" Test Connection

The NarrativeQA long setting directly inspired the "Needle in a Haystack" evaluation (Kamradt, 2023) — placing a specific fact anywhere in a 100k-token document and testing whether the model can retrieve it. NarrativeQA long is the naturalistic version of this synthetic test.

NarrativeQA (Long) is consuming the novel — the frontier benchmark of truly long-form document comprehension, where genuine understanding requires reading and integrating an entire book rather than finding and extracting a relevant passage.