reference-based evaluation

**Reference-based evaluation** is an assessment approach where a model's output is compared against a **gold standard reference answer** — a human-written or expert-verified "correct" response. The similarity between the model output and the reference determines the evaluation score. **Common Reference-Based Metrics** - **BLEU (Bilingual Evaluation Understudy)**: Measures **n-gram overlap** between the candidate and reference. Originally designed for machine translation. Score range 0–1. - **ROUGE (Recall-Oriented Understudy for Gisting Evaluation)**: Measures recall-oriented n-gram overlap. Widely used for **summarization** evaluation (ROUGE-1, ROUGE-2, ROUGE-L). - **METEOR**: Considers **synonyms**, **stemming**, and **word order** beyond simple n-gram matching. More nuanced than BLEU. - **BERTScore**: Uses **contextual embeddings** from BERT to compute semantic similarity between tokens, capturing meaning beyond surface-level word matching. - **Exact Match (EM)**: Binary — does the output exactly match the reference? Used for QA tasks with short, definitive answers. **Advantages** - **Automated and Fast**: No human judges needed once references are created. Evaluations run in seconds. - **Reproducible**: Same references, same metric, same score every time. - **Well-Understood**: Decades of research on these metrics, with known strengths and limitations. **Limitations** - **Reference Dependency**: The evaluation is only as good as the reference. For open-ended tasks, there are **many valid responses** not captured by a single reference. - **Surface Matching**: Metrics like BLEU penalize valid paraphrases that use different words to express the same meaning. - **Creativity Penalty**: For creative writing and open-ended generation, diverging from the reference is **desirable** but gets penalized. - **Single Reference Problem**: Using one reference misses the diversity of valid answers. Multiple references help but are expensive to create. Reference-based evaluation works best for **constrained tasks** (translation, summarization, factual QA) and is less suitable for open-ended generation where reference-free evaluation is preferred.