Fleiss' Kappa is a statistical measure of inter-annotator agreement designed for situations where more than two raters independently categorize items into fixed categories. It extends Cohen's Kappa (which only handles two raters) to any number of annotators.

The Formula

Where:

• $\bar{P}$ = mean observed agreement — the average proportion of annotator pairs that agree on each item.
• $\bar{P}_e$ = mean expected agreement by chance — computed from the overall proportion of annotations in each category.

How It Differs from Cohen's Kappa

• Cohen's Kappa: Exactly 2 annotators who each label all items.
• Fleiss' Kappa: Any number of annotators, but each item must be rated by the same number of annotators (though which specific annotators can vary per item).

Example Scenario

10 annotators each label 100 headlines as "clickbait" or "legitimate." Each headline gets rated by all 10 annotators. Fleiss' Kappa measures how much the 10 annotators agree beyond what chance would predict.

Interpretation

Same scale as Cohen's Kappa:

• κ < 0.20: Poor agreement
• 0.21–0.40: Fair
• 0.41–0.60: Moderate
• 0.61–0.80: Substantial
• 0.81–1.00: Almost perfect

Practical Applications

• Crowdsourcing QA: Measure agreement among MTurk workers or other crowd annotators to assess data quality.
• Benchmark Validation: Verify that human evaluations of model outputs are reliable.
• Medical Diagnosis: Multiple doctors rating the same cases to establish diagnostic reliability.

Limitations

• Fixed Number of Raters per Item: Each item must be rated by the same number of annotators (use Krippendorff's Alpha if this varies).
• Nominal Data Only: Designed for categorical labels. For ordinal or continuous data, use other metrics.
• Prevalence Sensitivity: Like Cohen's Kappa, can be artificially low when one category dominates.

Fleiss' Kappa is the standard choice for measuring agreement in multi-annotator labeling tasks, widely used in NLP dataset creation and evaluation.