inter-annotator agreement

**Inter-annotator agreement (IAA)** measures how consistently **multiple human evaluators** assign the same labels or scores to the same data. It is a critical quality metric for any dataset, benchmark, or evaluation process that relies on human judgment. **Why IAA Matters** - **Data Quality Signal**: Low agreement suggests the task is poorly defined, guidelines are unclear, or the task is inherently ambiguous. - **Upper Bound on ML Performance**: If humans can't agree on the correct label, a machine learning model trained on that data has an inherent ceiling on achievable accuracy. - **Evaluation Validity**: Benchmarks with low IAA produce unreliable rankings — random variation in labels means model comparisons are noisy. **Common IAA Metrics** - **Percent Agreement**: Simply the fraction of examples where annotators agree. Easy to compute but **doesn't account for chance** agreement. - **Cohen's Kappa (κ)**: Measures agreement between **two annotators**, correcting for chance agreement. Values: 0 = chance, 1 = perfect agreement. - **Fleiss' Kappa**: Extends Cohen's Kappa to **more than two annotators**. - **Krippendorff's Alpha**: Most general — handles multiple annotators, missing data, and various measurement scales (nominal, ordinal, interval, ratio). **Interpretation Guidelines** (Landis & Koch) - **κ < 0.20**: Poor agreement - **0.21–0.40**: Fair agreement - **0.41–0.60**: Moderate agreement - **0.61–0.80**: Substantial agreement - **0.81–1.00**: Almost perfect agreement **Best Practices** - **Pilot Annotation**: Have a small group annotate the same examples first, measure IAA, and refine guidelines before large-scale annotation. - **Calibration Sessions**: Regular meetings where annotators discuss disagreements and align their interpretation of guidelines. - **Adjudication**: For low-agreement examples, have a senior annotator or committee make the final decision. IAA should be **reported in every paper** that introduces a new dataset or evaluation — it quantifies the reliability ceiling of the human labels.