Inter-annotator agreement (IAA) measures how consistently multiple human evaluators assign the same labels or scores to the same data. It is a critical quality metric for any dataset, benchmark, or evaluation process that relies on human judgment.

Why IAA Matters

• Data Quality Signal: Low agreement suggests the task is poorly defined, guidelines are unclear, or the task is inherently ambiguous.
• Upper Bound on ML Performance: If humans can't agree on the correct label, a machine learning model trained on that data has an inherent ceiling on achievable accuracy.
• Evaluation Validity: Benchmarks with low IAA produce unreliable rankings — random variation in labels means model comparisons are noisy.

Common IAA Metrics

• Percent Agreement: Simply the fraction of examples where annotators agree. Easy to compute but doesn't account for chance agreement.
• Cohen's Kappa (κ): Measures agreement between two annotators, correcting for chance agreement. Values: 0 = chance, 1 = perfect agreement.
• Fleiss' Kappa: Extends Cohen's Kappa to more than two annotators.
• Krippendorff's Alpha: Most general — handles multiple annotators, missing data, and various measurement scales (nominal, ordinal, interval, ratio).

Interpretation Guidelines (Landis & Koch)

• κ < 0.20: Poor agreement
• 0.21–0.40: Fair agreement
• 0.41–0.60: Moderate agreement
• 0.61–0.80: Substantial agreement
• 0.81–1.00: Almost perfect agreement

Best Practices

• Pilot Annotation: Have a small group annotate the same examples first, measure IAA, and refine guidelines before large-scale annotation.
• Calibration Sessions: Regular meetings where annotators discuss disagreements and align their interpretation of guidelines.
• Adjudication: For low-agreement examples, have a senior annotator or committee make the final decision.

IAA should be reported in every paper that introduces a new dataset or evaluation — it quantifies the reliability ceiling of the human labels.