cohen's kappa
**Cohen's Kappa (κ)** is a statistical measure of **inter-annotator agreement** between **two raters** that corrects for the amount of agreement expected by **random chance**. It is one of the most widely used metrics for assessing the reliability of human annotations in NLP and machine learning.
**The Formula**
$$\kappa = \frac{p_o - p_e}{1 - p_e}$$
Where:
- $p_o$ = **observed agreement** — the proportion of items where both annotators assigned the same label.
- $p_e$ = **expected agreement by chance** — the agreement that would occur if annotators labeled randomly according to their marginal label distributions.
**Example Calculation**
Two annotators label 100 movie reviews as positive or negative:
- They agree on 85 reviews ($p_o = 0.85$)
- By chance alone, they'd agree on about 52 ($p_e = 0.52$)
- $\kappa = (0.85 - 0.52)/(1 - 0.52) = 0.33/0.48 = 0.69$ — substantial agreement
**Interpretation**
- **κ = 1**: Perfect agreement
- **κ = 0**: Agreement is no better than chance
- **κ < 0**: Agreement is worse than chance (systematic disagreement)
- **κ > 0.8**: Generally considered excellent for most NLP tasks
**Strengths**
- **Chance Correction**: Unlike raw percent agreement, Kappa recognizes that some agreement happens by luck.
- **Widely Understood**: Standard metric across NLP, medicine, psychology, and social sciences.
- **Easy to Compute**: Simple formula with readily available implementations.
**Limitations**
- **Two Raters Only**: Cohen's Kappa works for exactly two annotators. For more, use **Fleiss' Kappa** or **Krippendorff's Alpha**.
- **Nominal Data Only**: Designed for categorical labels. For ordinal data, use **weighted Kappa**.
- **Prevalence Sensitivity**: When one category is much more common, high raw agreement can still yield low Kappa due to high expected chance agreement (the **Kappa paradox**).