Krippendorff's Alpha (α) is a versatile reliability coefficient for measuring inter-annotator agreement that handles any number of annotators, any measurement scale, and missing data. Developed by Klaus Krippendorff, it is considered the most general and robust agreement metric available.

The Formula

Where:

• $D_o$ = observed disagreement — the actual disagreement in the data.
• $D_e$ = expected disagreement — the disagreement expected if annotations were random.

Key Advantages

• Any Number of Annotators: Works with 2, 3, 10, or 100 raters — no need for separate formulas.
• Missing Data: Unlike Cohen's Kappa, it handles situations where not every annotator labels every item.
• Multiple Measurement Scales: Supports nominal (categories), ordinal (ranked), interval (numeric with meaningful differences), and ratio (numeric with meaningful zero) data.
• Scale-Appropriate Distance: Uses a distance function matched to the measurement scale — nominal uses binary match/mismatch, ordinal uses rank differences, interval uses squared differences.

Interpretation

• α = 1: Perfect agreement
• α = 0: Agreement equals chance expectation
• α < 0: Systematic disagreement
• Krippendorff recommends: α ≥ 0.80 for reliable conclusions, α ≥ 0.667 for tentative conclusions, below 0.667 data should be discarded.

When to Use Krippendorff's Alpha

• When you have more than two annotators (unlike Cohen's Kappa)
• When not all annotators label every item (common in crowdsourcing)
• When data is ordinal or continuous (Cohen's Kappa only handles nominal)
• When you want a single, unified metric across different annotation projects

Implementations

Available in NLTK, scikit-learn, R (irr package), and standalone Python libraries like krippendorff.

Krippendorff's Alpha is increasingly recommended as the default agreement metric for annotation projects due to its generality and robustness.