Human Evaluation for LLMs

Why Human Evaluation? Automated metrics miss nuances that humans catch: creativity, helpfulness, safety, and overall quality.

Evaluation Types

Annotation Platforms

MTurk Setup

import boto3

mturk = boto3.client("mturk",
    region_name="us-east-1",
    endpoint_url="https://mturk-requester.us-east-1.amazonaws.com"
)

# Create HIT
response = mturk.create_hit(
    Title="Evaluate AI Response",
    Description="Rate the quality of AI responses",
    Keywords="AI, evaluation, rating",
    Reward="0.10",
    MaxAssignments=5,
    LifetimeInSeconds=86400,
    Question=open("eval_template.xml").read()
)

Evaluation Template

<QuestionForm>
  <Question>
    <QuestionContent>
      <Text>Rate this AI response:</Text>
      <Text>[Response here]</Text>
    </QuestionContent>
    <AnswerSpecification>
      <SelectionAnswer>
        <StyleSuggestion>radiobutton</StyleSuggestion>
        <Selections>
          <Selection><SelectionIdentifier>1</SelectionIdentifier>
            <Text>Very Poor</Text></Selection>
          <Selection><SelectionIdentifier>5</SelectionIdentifier>
            <Text>Excellent</Text></Selection>
        </Selections>
      </SelectionAnswer>
    </AnswerSpecification>
  </Question>
</QuestionForm>

Inter-Annotator Agreement

from sklearn.metrics import cohen_kappa_score

# Measure agreement between annotators
kappa = cohen_kappa_score(annotator1_ratings, annotator2_ratings)
# kappa > 0.8: strong agreement
# kappa 0.6-0.8: substantial
# kappa 0.4-0.6: moderate

Quality Control

Best Practices

• Clear, detailed instructions
• Use multiple annotators (3-5)
• Include quality control items
• Pay fairly for quality work
• Measure inter-annotator agreement