fleiss' kappa
**Fleiss' Kappa** is a statistical measure of **inter-annotator agreement** designed for situations where **more than two raters** independently categorize items into fixed categories. It extends Cohen's Kappa (which only handles two raters) to any number of annotators.
**The Formula**
$$\kappa = \frac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e}$$
Where:
- $\bar{P}$ = **mean observed agreement** — the average proportion of annotator pairs that agree on each item.
- $\bar{P}_e$ = **mean expected agreement by chance** — computed from the overall proportion of annotations in each category.
**How It Differs from Cohen's Kappa**
- **Cohen's Kappa**: Exactly **2 annotators** who each label **all items**.
- **Fleiss' Kappa**: **Any number of annotators**, but each item must be rated by the **same number** of annotators (though which specific annotators can vary per item).
**Example Scenario**
10 annotators each label 100 headlines as "clickbait" or "legitimate." Each headline gets rated by all 10 annotators. Fleiss' Kappa measures how much the 10 annotators agree beyond what chance would predict.
**Interpretation**
Same scale as Cohen's Kappa:
- **κ < 0.20**: Poor agreement
- **0.21–0.40**: Fair
- **0.41–0.60**: Moderate
- **0.61–0.80**: Substantial
- **0.81–1.00**: Almost perfect
**Practical Applications**
- **Crowdsourcing QA**: Measure agreement among MTurk workers or other crowd annotators to assess data quality.
- **Benchmark Validation**: Verify that human evaluations of model outputs are reliable.
- **Medical Diagnosis**: Multiple doctors rating the same cases to establish diagnostic reliability.
**Limitations**
- **Fixed Number of Raters per Item**: Each item must be rated by the same number of annotators (use **Krippendorff's Alpha** if this varies).
- **Nominal Data Only**: Designed for categorical labels. For ordinal or continuous data, use other metrics.
- **Prevalence Sensitivity**: Like Cohen's Kappa, can be artificially low when one category dominates.
Fleiss' Kappa is the standard choice for measuring agreement in **multi-annotator** labeling tasks, widely used in NLP dataset creation and evaluation.