meteor

METEOR (Metric for Evaluation of Translation with Explicit ORdering) is an evaluation metric for machine translation and text generation that addresses several limitations of BLEU by incorporating synonyms, stemming, paraphrase matching, and word order assessment to more closely correlate with human translation quality judgments. Introduced by Banerjee and Lavie in 2005, METEOR was designed to achieve better correlation with human judgments at both the segment level (individual sentences) and corpus level. METEOR computes a score through multi-stage alignment and scoring: first, it creates a word-by-word alignment between the candidate and reference using three modules applied sequentially — exact matching (identical surface forms), stemming (matching words sharing the same stem — "running" matches "runs" via Porter stemmer), and synonym matching (using WordNet synsets — "big" matches "large"). The best alignment maximizing matched words is selected. From this alignment, METEOR computes unigram precision (P = matched/candidate_length) and unigram recall (R = matched/reference_length), combined into a parameterized F-measure heavily weighted toward recall: F = (P × R) / (α × P + (1-α) × R), with α = 0.9 giving approximately 9× weight to recall over precision. A fragmentation penalty reduces the score when matched words are not in contiguous chunks — more chunks (worse word order) yields higher penalty: Penalty = γ × (chunks/matches)^β, with default γ=0.5, β=3. Final score: METEOR = F × (1 - Penalty). Key advantages over BLEU include: meaningful sentence-level scores (BLEU is unreliable for individual sentences), synonym and stem matching (capturing semantic equivalence beyond surface forms), explicit word order evaluation (through the fragmentation penalty), and consistently higher correlation with human judgments in evaluation campaigns. METEOR has been extended with paraphrase tables for broader matching coverage and tunable parameters for different languages and tasks. While computationally more expensive than BLEU due to alignment and WordNet lookups, METEOR remains widely used alongside BLEU and newer model-based metrics.