krippendorff's alpha

**Krippendorff's Alpha (α)** is a versatile reliability coefficient for measuring **inter-annotator agreement** that handles any number of annotators, any measurement scale, and missing data. Developed by Klaus Krippendorff, it is considered the most general and robust agreement metric available. **The Formula** $$\alpha = 1 - \frac{D_o}{D_e}$$ Where: - $D_o$ = **observed disagreement** — the actual disagreement in the data. - $D_e$ = **expected disagreement** — the disagreement expected if annotations were random. **Key Advantages** - **Any Number of Annotators**: Works with 2, 3, 10, or 100 raters — no need for separate formulas. - **Missing Data**: Unlike Cohen's Kappa, it handles situations where not every annotator labels every item. - **Multiple Measurement Scales**: Supports **nominal** (categories), **ordinal** (ranked), **interval** (numeric with meaningful differences), and **ratio** (numeric with meaningful zero) data. - **Scale-Appropriate Distance**: Uses a distance function matched to the measurement scale — nominal uses binary match/mismatch, ordinal uses rank differences, interval uses squared differences. **Interpretation** - **α = 1**: Perfect agreement - **α = 0**: Agreement equals chance expectation - **α < 0**: Systematic disagreement - Krippendorff recommends: **α ≥ 0.80** for reliable conclusions, **α ≥ 0.667** for tentative conclusions, below 0.667 data should be discarded. **When to Use Krippendorff's Alpha** - When you have **more than two annotators** (unlike Cohen's Kappa) - When **not all annotators label every item** (common in crowdsourcing) - When data is **ordinal or continuous** (Cohen's Kappa only handles nominal) - When you want a **single, unified metric** across different annotation projects **Implementations** Available in **NLTK**, **scikit-learn**, **R (irr package)**, and standalone Python libraries like **krippendorff**. Krippendorff's Alpha is increasingly recommended as the **default agreement metric** for annotation projects due to its generality and robustness.