annotator disagreement

**Annotator disagreement** occurs when multiple human labelers assign **different labels** to the same data example. Understanding and managing disagreement is crucial because it directly impacts the quality of training data and the reliability of evaluation benchmarks. **Sources of Disagreement** - **Genuine Ambiguity**: The example is inherently ambiguous — reasonable people can legitimately disagree. "This movie was interesting" — positive or neutral sentiment? - **Unclear Guidelines**: Annotation instructions don't cover the specific case or are interpreted differently by different annotators. - **Annotator Error**: Mistakes due to fatigue, carelessness, or misunderstanding of the task. - **Subjectivity**: Tasks involving judgment calls (toxicity, quality, humor) naturally produce more disagreement than factual tasks. - **Cultural Differences**: Annotators from different backgrounds may interpret the same content differently. **How to Handle Disagreement** - **Majority Vote**: Use the label chosen by the majority of annotators. Simple but loses information about uncertainty. - **Adjudication**: A senior annotator or expert reviews disagreements and makes the final decision. - **Probabilistic Labels**: Instead of a single label, keep the **distribution of annotator votes** as a soft label (e.g., 60% positive, 40% neutral). - **Discard Ambiguous Examples**: Remove examples with low agreement from the dataset. Reduces noise but may bias the data. - **Model Disagreement**: If trained on data where annotators agree, models may not handle genuinely ambiguous real-world cases well. **Measuring Disagreement** - **Inter-Annotator Agreement**: Cohen's κ, Fleiss' κ, Krippendorff's α quantify overall consistency. - **Per-Example Agreement**: Some examples have 100% agreement, others have 50/50 splits. Analyzing the distribution reveals systematic patterns. **Modern Perspective** Recent research argues that disagreement is often **informative, not noise**. The field is moving toward **learning from disagreement** — training models that output calibrated uncertainty rather than forcing a single label. This is especially important for subjective tasks like toxicity detection, sentiment analysis, and content moderation.