wnli

**WNLI (Winograd NLI)** is a **conversion of the Winograd Schema Challenge into an NLI format, included in the GLUE benchmark** — arguably the most problematic task in GLUE due to its small size and distribution issues. **Problem** - **Structure**: Sentence A: "The trophy didn't fit in the suitcase because it was too large." Sentence B: "The trophy was too large." (Entailment). - **Difficulty**: The train/dev sets were disjoint in a way that made learning impossible for BERT. - **Hack**: The standard baseline was simply to predict "Majority Class" (65%), which BERT failed to beat for years. - **Outcome**: Generally ignored or treated as a "solved/broken" metric in GLUE analysis. **Why It Matters** - **Cautionary Tale**: Shows the difficulty of converting specific reasoning tasks (WSC) into generic formats (NLI). - **Winogrande**: This failure led to the creation of Winogrande (larger, robust WSC). **WNLI** is **the broken benchmark** — a well-intentioned attempt to include Winograd schemas in GLUE that ended up verifying very little.