reward model
**Reward Models and Preference Learning**
**What is a Reward Model?**
A model trained to predict human preferences, used to guide LLM training via RLHF.
**Preference Data Collection**
```
Prompt: "Explain photosynthesis"
Response A: [detailed explanation]
Response B: [brief explanation]
Human preference: A > B (A is better)
```
**Training Reward Model**
The reward model learns from pairwise comparisons:
```python
class RewardModel(nn.Module):
def __init__(self, base_model):
super().__init__()
self.backbone = base_model
self.reward_head = nn.Linear(hidden_size, 1)
def forward(self, input_ids):
hidden = self.backbone(input_ids).last_hidden_state[:, -1]
return self.reward_head(hidden)
# Bradley-Terry loss for pairwise preferences
def preference_loss(reward_chosen, reward_rejected):
return -torch.log(torch.sigmoid(reward_chosen - reward_rejected))
```
**Data Collection Methods**
| Method | Description |
|--------|-------------|
| Pairwise comparison | A vs B, which is better |
| Rating scale | Rate 1-5 |
| Ranking | Order multiple responses |
| Best-of-N | Pick best from N options |
**Reward Model Training**
```python
# Training loop
for batch in dataloader:
chosen = batch["chosen"] # Preferred response
rejected = batch["rejected"] # Less preferred
r_chosen = reward_model(chosen)
r_rejected = reward_model(rejected)
loss = preference_loss(r_chosen, r_rejected)
loss.backward()
optimizer.step()
```
**Using Reward Model in RLHF**
```
1. Generate response from LLM
2. Score with reward model
3. Use score as RL reward
4. Update LLM with PPO
```
**Challenges**
| Challenge | Mitigation |
|-----------|------------|
| Reward hacking | Regularize, diverse prompts |
| Annotation quality | Multiple annotators, guidelines |
| Distribution shift | Retrain on new model outputs |
| Mode collapse | KL penalty to reference model |
**DPO Alternative**
Direct Preference Optimization skips explicit reward model:
```python
# DPO loss (simplified)
log_ratio_chosen = log_prob_policy(chosen) - log_prob_ref(chosen)
log_ratio_rejected = log_prob_policy(rejected) - log_prob_ref(rejected)
loss = -log_sigmoid(beta * (log_ratio_chosen - log_ratio_rejected))
```
**Best Practices**
- Collect high-quality preference data
- Train on diverse prompts
- Monitor for reward hacking
- Combine with other alignment techniques
- Iterate on annotation guidelines