Cohen's Kappa (κ) is a statistical measure of inter-annotator agreement between two raters that corrects for the amount of agreement expected by random chance. It is one of the most widely used metrics for assessing the reliability of human annotations in NLP and machine learning.

The Formula

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Where:

• $p_o$ = observed agreement — the proportion of items where both annotators assigned the same label.
• $p_e$ = expected agreement by chance — the agreement that would occur if annotators labeled randomly according to their marginal label distributions.

Example Calculation

Two annotators label 100 movie reviews as positive or negative:

• They agree on 85 reviews ($p_o = 0.85$)
• By chance alone, they'd agree on about 52 ($p_e = 0.52$)
• $\kappa = (0.85 - 0.52)/(1 - 0.52) = 0.33/0.48 = 0.69$ — substantial agreement

Interpretation

• κ = 1: Perfect agreement
• κ = 0: Agreement is no better than chance
• κ < 0: Agreement is worse than chance (systematic disagreement)
• κ > 0.8: Generally considered excellent for most NLP tasks

Strengths

• Chance Correction: Unlike raw percent agreement, Kappa recognizes that some agreement happens by luck.
• Widely Understood: Standard metric across NLP, medicine, psychology, and social sciences.
• Easy to Compute: Simple formula with readily available implementations.

Limitations

• Two Raters Only: Cohen's Kappa works for exactly two annotators. For more, use Fleiss' Kappa or Krippendorff's Alpha.
• Nominal Data Only: Designed for categorical labels. For ordinal data, use weighted Kappa.
• Prevalence Sensitivity: When one category is much more common, high raw agreement can still yield low Kappa due to high expected chance agreement (the Kappa paradox).