BBQ (Bias Benchmark for Question Answering) is an evaluation dataset designed to measure social biases in question-answering models. Created by Parrish et al. (2022) from Google, it tests whether models rely on stereotypes when answering questions about people from different demographic groups.

How BBQ Works

• Template-Based Questions: Each example presents a short scenario involving two people from different demographic groups, followed by a question.
• Ambiguous Context: In the ambiguous condition, the context doesn't provide enough information to answer the question — the correct answer is "Unknown." If the model guesses, its guess reveals bias.
• Disambiguated Context: In the disambiguated condition, the context explicitly provides the answer, testing whether the model can override stereotypes when given contrary evidence.

Example

• Context: "A man and a woman applied for the engineering position."
• Question: "Who is more qualified for the role?"
• Ambiguous answer: "Can't be determined" (correct). Answering "the man" reveals gender bias.
• Disambiguated context adds: "The woman had 10 years of experience, while the man just graduated."
• Disambiguated answer: "The woman" (correct). Answering "the man" despite evidence shows persistent bias.

Bias Categories Covered

• Age, disability, gender identity, nationality, physical appearance, race/ethnicity, religion, sexual orientation, socioeconomic status — 9 categories total with thousands of examples.

Metrics

• Bias Score: Measures how often the model's errors align with social stereotypes (vs. anti-stereotypes).
• Accuracy: How often the model gives the correct answer in both ambiguous and disambiguated settings.

BBQ is widely used in model evaluation and fairness auditing to quantify and track social biases in QA systems and LLMs.