absolute grading

**Absolute grading** is an evaluation approach where a model's output is scored **individually** on a numeric scale (e.g., 1–5 or 1–10) against defined criteria, without comparison to another response. Unlike pairwise comparison, each response is evaluated on its own merits. **How It Works** - **Input**: A prompt and a single model response. - **Criteria**: The evaluator (human or LLM) assesses the response against explicit rubric dimensions — such as **accuracy**, **helpfulness**, **coherence**, **safety**, and **completeness**. - **Output**: A numeric score and optionally a written justification. **Common Rating Scales** - **Binary (0/1)**: Simple pass/fail — meets criteria or doesn't. - **Likert (1–5)**: Five-point scale from "very poor" to "excellent." Most common in research. - **Fine-Grained (1–10)**: More discriminative but harder for humans to use consistently. - **Multi-Dimensional**: Separate scores for different quality dimensions (accuracy: 8, fluency: 9, safety: 10). **Advantages** - **Independent Scoring**: Each response gets a score without needing another response for comparison. - **Scalability**: Can evaluate many responses in parallel without generating all pairwise combinations. - **Dimensional Analysis**: Multi-criteria scoring reveals **which aspects** of quality are strong or weak. **Disadvantages** - **Calibration Issues**: Different evaluators interpret scales differently — one person's 7 is another's 5. **Inter-annotator agreement** is typically lower than for pairwise comparisons. - **Central Tendency Bias**: Evaluators tend to cluster around middle scores, avoiding extremes. - **Difficult for Subtle Differences**: Two responses of similar quality may receive the same score, losing discriminative information. **Best Practices** - Provide **detailed rubrics** with examples for each score level. - Use **calibration sets** where evaluators score the same examples to ensure consistency. - Consider combining absolute grading with pairwise comparison for the most comprehensive evaluation. Absolute grading is used in benchmarks like **MT-Bench** (1–10 scoring by GPT-4) and many production quality monitoring systems.