QuALITY (Question Answering with Long Input Texts, Yes!) is the multiple-choice QA benchmark specifically designed to require reading and reasoning over the entire 5,000-token document — with distractors carefully crafted to be plausible for readers who skimmed the text, explicitly adversarial against RAG and chunk-retrieval approaches, and validated through a speed-controlled annotation process that ensures questions cannot be answered without full reading comprehension.

What Is QuALITY?

• Origin: Pang et al. (2022).
• Scale: 2,523 multiple-choice questions over 233 articles/stories, averaging 5,000 tokens per document.
• Format: 4-option multiple-choice; one correct answer requires whole-document understanding.
• Sources: Fiction from Project Gutenberg and science fiction magazines (Tor, Clarkesworld); non-fiction articles on science and society.
• Annotation: Human annotators had to read the full document before writing questions — and crucially, the annotation interface measured reading speed to verify comprehension.

The Anti-RAG Design

QuALITY was deliberately engineered to defeat retrieval-based shortcuts:

• Global Synthesis Questions: "What was the protagonist's primary motivation throughout the story?" — requires integrating character intentions from beginning, middle, and end.
• Contrast Questions: "Which of the following events occurred but did NOT influence the climax?" — requires knowing what events did and did not occur throughout the entire narrative.
• Negation Across Sections: "Which character was NOT present at both the opening ceremony and the final confrontation?" — requires tracking presence/absence across the full document.
• Plausible Distractors: Wrong answers are facts from the text that appear relevant if you didn't read everything — they cannot be eliminated by finding a single relevant passage.

Speed Annotation Validation

A key QuALITY innovation is annotator speed validation:

• Annotators who completed the annotation too quickly (implying skimming) were flagged and their questions reviewed.
• Only questions from annotators who demonstrably read the full text were included.
• This prevents the dataset from containing questions answerable from summaries or abstracts.

Performance Results

The RAG Gap

Comparing lexical retrieval (~42%) to full-context GPT-4 (71.6%) demonstrates the ~30-point accuracy gap of chunk-retrieval approaches on QuALITY — the largest documented accuracy gap anywhere in long-document QA benchmarks.

Why QuALITY Matters

• RAG Limitation Quantification: QuALITY provides the clearest evidence that RAG-based systems have systematic blind spots for questions requiring global document understanding.
• Context Window Validation: Every extension of commercial LLM context windows (from 4k to 128k) should demonstrate improvement on QuALITY to justify the computational cost.
• Reading Comprehension Benchmark: QuALITY is the most rigorous test of genuine reading comprehension — it measures what humans mean when they say "read the document," not "scan for the relevant sentence."
• Question Quality: The annotator-speed-filtered questions are among the highest quality in NLP benchmarks — very few annotation errors compared to crowdsourced datasets.
• Cost-Accuracy Trade-off: For legal and medical applications, knowing that full-context models are 30 points better than RAG on global questions directly informs architecture choices despite higher inference cost.

Comparison to Related Long-Context Benchmarks

QuALITY is deep reading for AI — the benchmark that proves whether language models genuinely read and synthesize entire documents or merely locate and extract relevant passages, with deliberately adversarial question design that quantifies the comprehension gap between retrieval shortcuts and true long-form reading comprehension.