winogender

**WinoGender** is a diagnostic evaluation dataset designed to test **gender bias** in **coreference resolution** systems — specifically, whether models rely on **occupational stereotypes** when determining who a pronoun refers to. **How WinoGender Works** - **Sentence Template**: Each example contains two people (identified by occupation) and a pronoun that refers to one of them. - **Stereotype Testing**: One occupation is stereotypically male (e.g., mechanic), another stereotypically female (e.g., nurse), and the correct referent is varied to test whether models follow stereotypes. **Example Pairs** - "**The mechanic** called **the nurse** because **he** needed help." → "he" = mechanic (stereotype-consistent) - "**The mechanic** called **the nurse** because **he** was running late." → "he" = nurse (stereotype-inconsistent: nurse referred to as "he") - An unbiased model should resolve both correctly based on **context**, not occupation stereotypes. **Key Design Features** - **720 Sentence Pairs**: Covering 60 occupations from Bureau of Labor Statistics data with real-world gender composition statistics. - **Three Pronoun Conditions**: Male ("he/him"), female ("she/her"), and neutral ("they/them") versions of each template. - **Matched Structure**: Sentences are identical except for the pronoun and which entity it refers to, isolating the effect of gender bias. **What WinoGender Reveals** - Models show **higher accuracy** when pronouns align with occupational stereotypes (e.g., "she" referring to a nurse, "he" referring to a doctor). - **Accuracy drops** significantly when pronouns contradict stereotypes (e.g., "he" referring to a nurse). - Performance gaps directly quantify the model's reliance on **gender stereotypes** rather than linguistic context. **Related Benchmarks**: **WinoBias** (similar concept, larger dataset), **WinoGrande** (general commonsense, not bias-specific), and **WinoMT** (bias in machine translation). WinoGender is referenced in major AI fairness papers and is part of standard **bias evaluation suites** for NLP models.