crows-pairs

**CrowS-Pairs** is the **fairness benchmark based on paired minimally different sentences that contrast stereotypical and anti-stereotypical statements** - it measures whether models assign higher likelihood to biased phrasing. **What Is CrowS-Pairs?** - **Definition**: Dataset of sentence pairs differing mainly in stereotype direction for protected groups. - **Evaluation Mechanism**: Compare model preference or pseudo-likelihood between paired sentences. - **Bias Dimensions**: Covers categories such as race, gender, religion, age, and disability. - **Metric Goal**: Lower stereotype-preference bias indicates fairer language modeling behavior. **Why CrowS-Pairs Matters** - **Fine-Grained Testing**: Minimal-pair setup isolates bias signal from unrelated content variation. - **Model Comparison**: Supports consistent fairness ranking across architectures and versions. - **Mitigation Validation**: Sensitive to changes from debiasing interventions. - **Interpretability**: Pairwise outcomes are easy to inspect for qualitative error analysis. - **Governance Support**: Useful for regression monitoring in release pipelines. **How It Is Used in Practice** - **Batch Scoring**: Evaluate model likelihood preference across full pair set by subgroup. - **Disparity Breakdown**: Report results by protected category to localize weaknesses. - **Integrated Review**: Use with complementary benchmarks to avoid single-metric blind spots. CrowS-Pairs is **a widely used minimal-pair fairness benchmark for LLMs** - pairwise stereotype preference testing provides clear, actionable bias diagnostics for model evaluation workflows.