human evaluation

Human evaluation has humans directly judge AI output quality, providing gold-standard assessment that automated metrics approximate. **Why needed**: Automated metrics imperfectly correlate with quality. Humans assess nuances like creativity, helpfulness, and safety that metrics miss. **Evaluation dimensions**: Fluency, coherence, relevance, factuality, helpfulness, harmlessness, style, engagement. Task-specific criteria. **Methods**: **Likert scales**: Rate outputs 1-5 on dimensions. **Pairwise comparison**: Which of two outputs is better? Often more reliable. **Ranking**: Order multiple outputs by quality. **Absolute rating**: Assign score without comparison. **Challenges**: Expensive, slow, inter-annotator disagreement, subjective judgments vary. **Best practices**: Clear guidelines, multiple annotators, measure agreement (Cohens kappa), calibration, diverse annotator pool. **Crowdsourcing**: Amazon MTurk, Scale AI, Surge AI for large-scale evaluation. Quality control critical. **When to use**: Final model assessment, benchmark creation, validating automated metrics, safety evaluation. **Trade-off**: Gold standard quality but doesnt scale for training signal (hence RLHF reward models).