bbq (bias benchmark for qa)

**BBQ (Bias Benchmark for Question Answering)** is an evaluation dataset designed to measure **social biases** in question-answering models. Created by Parrish et al. (2022) from Google, it tests whether models rely on **stereotypes** when answering questions about people from different demographic groups. **How BBQ Works** - **Template-Based Questions**: Each example presents a short scenario involving two people from different demographic groups, followed by a question. - **Ambiguous Context**: In the **ambiguous** condition, the context doesn't provide enough information to answer the question — the correct answer is "Unknown." If the model guesses, its guess reveals bias. - **Disambiguated Context**: In the **disambiguated** condition, the context explicitly provides the answer, testing whether the model can override stereotypes when given contrary evidence. **Example** - Context: "A man and a woman applied for the engineering position." - Question: "Who is more qualified for the role?" - Ambiguous answer: **"Can't be determined"** (correct). Answering "the man" reveals gender bias. - Disambiguated context adds: "The woman had 10 years of experience, while the man just graduated." - Disambiguated answer: **"The woman"** (correct). Answering "the man" despite evidence shows persistent bias. **Bias Categories Covered** - **Age**, **disability**, **gender identity**, **nationality**, **physical appearance**, **race/ethnicity**, **religion**, **sexual orientation**, **socioeconomic status** — 9 categories total with thousands of examples. **Metrics** - **Bias Score**: Measures how often the model's errors align with social stereotypes (vs. anti-stereotypes). - **Accuracy**: How often the model gives the correct answer in both ambiguous and disambiguated settings. BBQ is widely used in **model evaluation** and **fairness auditing** to quantify and track social biases in QA systems and LLMs.