Adversarial NLI (ANLI) is a difficult NLI benchmark created through human-in-the-loop adversarial data collection — examples that humans create specifically to fool state-of-the-art models, testing robust language understanding.

What Is Adversarial NLI?

• Type: Natural Language Inference benchmark.
• Method: Humans write premise-hypothesis pairs that fool models.
• Difficulty: Much harder than standard NLI datasets.
• Rounds: Three rounds of increasingly difficult examples.
• Purpose: Test model robustness and reasoning depth.

Why ANLI Matters

• Robustness: Exposes model weaknesses and shortcuts.
• Harder: Models that ace SNLI/MultiNLI struggle on ANLI.
• Iterative: Each round targets remaining model failures.
• Real Reasoning: Requires genuine understanding, not shortcuts.
• Research Standard: Used to evaluate robust NLU models.

Collection Process

1. Human sees premise and model prediction. 2. Human writes hypothesis to fool the model. 3. If model fails, example added to dataset. 4. Repeat with improved models for harder rounds.

Performance Gap

• BERT on MultiNLI: ~86%
• BERT on ANLI: ~45% (near random)

ANLI is the stress test for language understanding — exposing shortcuts models learn.