Annotator disagreement occurs when multiple human labelers assign different labels to the same data example. Understanding and managing disagreement is crucial because it directly impacts the quality of training data and the reliability of evaluation benchmarks.

Sources of Disagreement

• Genuine Ambiguity: The example is inherently ambiguous — reasonable people can legitimately disagree. "This movie was interesting" — positive or neutral sentiment?
• Unclear Guidelines: Annotation instructions don't cover the specific case or are interpreted differently by different annotators.
• Annotator Error: Mistakes due to fatigue, carelessness, or misunderstanding of the task.
• Subjectivity: Tasks involving judgment calls (toxicity, quality, humor) naturally produce more disagreement than factual tasks.
• Cultural Differences: Annotators from different backgrounds may interpret the same content differently.

How to Handle Disagreement

• Majority Vote: Use the label chosen by the majority of annotators. Simple but loses information about uncertainty.
• Adjudication: A senior annotator or expert reviews disagreements and makes the final decision.
• Probabilistic Labels: Instead of a single label, keep the distribution of annotator votes as a soft label (e.g., 60% positive, 40% neutral).
• Discard Ambiguous Examples: Remove examples with low agreement from the dataset. Reduces noise but may bias the data.
• Model Disagreement: If trained on data where annotators agree, models may not handle genuinely ambiguous real-world cases well.

Measuring Disagreement

• Inter-Annotator Agreement: Cohen's κ, Fleiss' κ, Krippendorff's α quantify overall consistency.
• Per-Example Agreement: Some examples have 100% agreement, others have 50/50 splits. Analyzing the distribution reveals systematic patterns.

Modern Perspective

Recent research argues that disagreement is often informative, not noise. The field is moving toward learning from disagreement — training models that output calibrated uncertainty rather than forcing a single label. This is especially important for subjective tasks like toxicity detection, sentiment analysis, and content moderation.