bleu score

BLEU (Bilingual Evaluation Understudy) is a precision-based automatic evaluation metric originally designed for machine translation quality assessment that measures the n-gram overlap between a candidate (generated) text and one or more reference (human-produced) texts. Introduced by Papineni et al. in 2002, BLEU became the standard metric for machine translation evaluation and has been widely adopted (and sometimes misapplied) across other text generation tasks including summarization, paraphrasing, and dialogue generation. BLEU computes modified precision for n-grams of different lengths (typically 1-grams through 4-grams): for each n-gram in the candidate, it checks whether that n-gram appears in any reference translation, with a clipping mechanism that limits matches to the maximum count of each n-gram across references (preventing artificially inflated scores from repeating common n-grams). The final BLEU score combines these n-gram precisions using a geometric mean with equal weights (typically BLEU-4 uses 1-gram through 4-gram precision), multiplied by a brevity penalty (BP) that penalizes translations shorter than the reference to prevent gaming the score with very short high-precision outputs: BP = min(1, exp(1 - reference_length/candidate_length)). BLEU ranges from 0 to 1 (often reported as 0-100), with higher scores indicating greater similarity to reference translations. Strengths include: language-independent (works for any language pair), fast computation, correlation with human judgments at the corpus level, and standardized implementation (SacreBLEU). Limitations include: poor correlation with human judgment at the sentence level, inability to capture meaning (semantically equivalent paraphrases may score poorly), insensitivity to word order beyond n-gram matching, bias toward shorter outputs (despite brevity penalty), and no accounting for synonyms or grammatical acceptability. Despite these limitations, BLEU remains widely reported as a baseline metric, though modern evaluation increasingly supplements it with model-based metrics like BERTScore, BLEURT, and COMET.