BBQ is the Bias Benchmark for Question Answering that evaluates social bias under both ambiguous and disambiguated context conditions - it tests whether models choose stereotyped answers when evidence is insufficient.

What Is BBQ?

• Definition: QA benchmark designed to measure biased response tendencies across social dimensions.
• Context Design: Includes ambiguous scenarios where correct answer should be unknown and clarified scenarios with explicit evidence.
• Bias Signal: Measures stereotype-consistent answer preference when uncertainty is present.
• Evaluation Output: Reports both accuracy and bias-related behavior metrics.

Why BBQ Matters

• Ambiguity Stress Test: Reveals whether models guess using stereotypes instead of abstaining.
• Fairness Diagnostics: Distinguishes true reasoning from socially biased shortcuts.
• Mitigation Benchmarking: Useful for assessing prompt and model debias interventions.
• Risk Relevance: QA systems are common in support and decision-assist applications.
• Governance Utility: Provides interpretable bias indicators for model release review.

How It Is Used in Practice

• Split Analysis: Evaluate performance separately on ambiguous and disambiguated subsets.
• Behavioral Metrics: Track stereotype-choice rates in uncertain contexts.
• Regression Tracking: Compare BBQ outcomes across model updates and alignment changes.

BBQ is an important fairness benchmark for QA behavior under uncertainty - it highlights whether models handle ambiguity responsibly or default to stereotype-based guessing.