bbq

**BBQ** is the **Bias Benchmark for Question Answering that evaluates social bias under both ambiguous and disambiguated context conditions** - it tests whether models choose stereotyped answers when evidence is insufficient. **What Is BBQ?** - **Definition**: QA benchmark designed to measure biased response tendencies across social dimensions. - **Context Design**: Includes ambiguous scenarios where correct answer should be unknown and clarified scenarios with explicit evidence. - **Bias Signal**: Measures stereotype-consistent answer preference when uncertainty is present. - **Evaluation Output**: Reports both accuracy and bias-related behavior metrics. **Why BBQ Matters** - **Ambiguity Stress Test**: Reveals whether models guess using stereotypes instead of abstaining. - **Fairness Diagnostics**: Distinguishes true reasoning from socially biased shortcuts. - **Mitigation Benchmarking**: Useful for assessing prompt and model debias interventions. - **Risk Relevance**: QA systems are common in support and decision-assist applications. - **Governance Utility**: Provides interpretable bias indicators for model release review. **How It Is Used in Practice** - **Split Analysis**: Evaluate performance separately on ambiguous and disambiguated subsets. - **Behavioral Metrics**: Track stereotype-choice rates in uncertain contexts. - **Regression Tracking**: Compare BBQ outcomes across model updates and alignment changes. BBQ is **an important fairness benchmark for QA behavior under uncertainty** - it highlights whether models handle ambiguity responsibly or default to stereotype-based guessing.