alignment
**LLM alignment** is the **process of training language models to behave in accordance with human values and intentions** — using techniques like RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) to make models helpful, harmless, and honest, ensuring AI systems do what users actually want rather than just predicting the next token.
**What Is Alignment?**
- **Definition**: Training AI to act according to human preferences and values.
- **Goal**: Models that are helpful, harmless, and honest (HHH).
- **Challenge**: Base models predict text, not "good" behavior.
- **Methods**: RLHF, DPO, Constitutional AI, instruction tuning.
**Why Alignment Matters**
- **Safety**: Prevent harmful, dangerous, or illegal outputs.
- **Usefulness**: Models should actually help with user tasks.
- **Trust**: Users must be able to rely on AI responses.
- **Control**: Aligned models follow instructions and boundaries.
- **Scaling**: Alignment must hold as models become more capable.
- **Existential**: Long-term AI safety depends on alignment.
**The Alignment Problem**
**Base Model Behavior**:
```
Prompt: "How do I pick a lock?"
Base Model (unaligned):
→ Has seen lockpicking instructions in training data
→ May helpfully provide detailed instructions
→ No concept of "should I answer this?"
Aligned Model:
→ Considers potential harm of response
→ May refuse or provide only legal context
→ Balances helpfulness with safety
```
**Alignment Methods**
**Supervised Fine-Tuning (SFT)**:
- Train on demonstrations of desired behavior.
- (Instruction, good-response) pairs.
- Shows what good responses look like.
- Foundation for further alignment.
**RLHF (Reinforcement Learning from Human Feedback)**:
```
Step 1: Collect comparisons
Prompt → Response A vs Response B
Human labels which is better
Step 2: Train reward model
Reward(prompt, response) → score
Predicts human preference
Step 3: Optimize policy
Use PPO to maximize reward
Policy = original model + value head
Iterate with fresh feedback
```
**DPO (Direct Preference Optimization)**:
```
Insight: Skip reward model, directly use preferences
Loss = -log σ(β × (log π(y_w|x)/π_ref(y_w|x)
- log π(y_l|x)/π_ref(y_l|x)))
y_w = preferred response
y_l = dis-preferred response
Simpler, often matches RLHF quality
```
**Constitutional AI (CAI)**:
```
1. Generate response to harmful prompt
2. Critique: "Does this response violate [principle]?"
3. Revise: "Write a response that doesn't..."
4. Fine-tune on revised responses
5. RLHF with AI feedback (RLAIF)
Principles: List of behavioral guidelines
Reduces need for human labeling
```
**Alignment Comparison**
```
Method | Human Data | Complexity | Quality
-------------|-------------|------------|----------
SFT | Demos | Simple | Baseline
RLHF | Comparisons | Complex | Best
DPO | Comparisons | Medium | Near RLHF
CAI/RLAIF | Principles | Medium | Good
```
**Challenges in Alignment**
- **Specification**: Hard to fully specify "human values."
- **Gaming**: Models can learn to satisfy reward without true alignment.
- **Distribution Shift**: Alignment may not generalize to new situations.
- **Scalability**: Alignment methods must scale with model capability.
- **Robustness**: Aligned models can still be jailbroken.
- **Cultural Variation**: Values differ across cultures.
**Current State**
- Modern chat models (ChatGPT, Claude, etc.) are heavily aligned.
- Alignment reduces raw capability in exchange for safety.
- Open models available in aligned and base versions.
- Active research on more robust alignment methods.
LLM alignment is **the critical challenge for beneficial AI** — getting powerful AI systems to reliably do what we want, avoid what we don't want, and behave ethically is essential for AI to be a positive force, making alignment research one of the most important areas in AI development.