llm as judge

**LLM As Judge** LLM-as-judge uses a strong language model to evaluate outputs from weaker models or different systems providing scalable automated evaluation. GPT-4 commonly serves as judge assessing quality correctness helpfulness and safety. This approach scales better than human evaluation while maintaining reasonable correlation with human judgments. Evaluation can be pairwise comparing two outputs pointwise scoring single outputs or reference-based comparing to gold standard. Prompts specify evaluation criteria rubrics and output format. Challenges include judge model biases like preferring its own outputs position bias favoring first option and verbosity bias preferring longer responses. Mitigation strategies include using multiple judges swapping comparison order and calibrating against human ratings. LLM-as-judge is valuable for iterative development A/B testing and continuous monitoring. It enables rapid experimentation when human evaluation is too slow or expensive. Limitations include inability to verify factual accuracy potential bias propagation and cost of API calls. Best practices include clear rubrics diverse test cases and periodic human validation.