nlvr (natural language for visual reasoning)

**NLVR** (Natural Language for Visual Reasoning) is a **benchmark task requiring models to determine the truth of a statement based on a *set* of images** — testing the ability to reason about properties, counts, and comparisons across multiple disjoint visual inputs. **What Is NLVR?** - **Definition**: Binary classification (True/False) of a sentence given a pair (or set) of images. - **Task**: "The left image contains exactly two dogs and the right image contains none." -> True/False. - **NLVR2**: The version using real web images (instead of synthetic ones) to test robustness. **Why NLVR Matters** - **Set Reasoning**: Unlike VQA (one image), NLVR requires holding information from Image A while analyzing Image B. - **Quantification**: Heavily tests counting and numerical comparison ("more than", "at least"). - **Robustness**: Reduces the ability to cheat using language biases alone. **NLVR** is **a test of comparative visual cognition** — validating that an AI can perform logical operations over multiple observations.