human eval
**Human Evaluation for LLMs**
**Why Human Evaluation?**
Automated metrics miss nuances that humans catch: creativity, helpfulness, safety, and overall quality.
**Evaluation Types**
| Type | What it Measures |
|------|------------------|
| Absolute rating | Rate response 1-5 |
| Pairwise comparison | A vs B, which is better? |
| Ranking | Order N responses |
| Task completion | Did it accomplish goal? |
| Aspect-based | Rate helpfulness, accuracy, etc. |
**Annotation Platforms**
| Platform | Type | Cost |
|----------|------|------|
| Amazon MTurk | Crowdsource | Low |
| Scale AI | Managed | High |
| Surge AI | Quality focus | Medium |
| Prolific | Academic | Medium |
| In-house | Expert | Variable |
**MTurk Setup**
```python
import boto3
mturk = boto3.client("mturk",
region_name="us-east-1",
endpoint_url="https://mturk-requester.us-east-1.amazonaws.com"
)
# Create HIT
response = mturk.create_hit(
Title="Evaluate AI Response",
Description="Rate the quality of AI responses",
Keywords="AI, evaluation, rating",
Reward="0.10",
MaxAssignments=5,
LifetimeInSeconds=86400,
Question=open("eval_template.xml").read()
)
```
**Evaluation Template**
```html
Rate this AI response:
[Response here]
radiobutton
1
Very Poor 5
Excellent
```
**Inter-Annotator Agreement**
```python
from sklearn.metrics import cohen_kappa_score
# Measure agreement between annotators
kappa = cohen_kappa_score(annotator1_ratings, annotator2_ratings)
# kappa > 0.8: strong agreement
# kappa 0.6-0.8: substantial
# kappa 0.4-0.6: moderate
```
**Quality Control**
| Method | Purpose |
|--------|---------|
| Gold questions | Catch low-effort workers |
| Redundancy | Multiple annotators per item |
| Qualification tests | Filter workers |
| Time limits | Prevent rushing |
**Best Practices**
- Clear, detailed instructions
- Use multiple annotators (3-5)
- Include quality control items
- Pay fairly for quality work
- Measure inter-annotator agreement