constitutional ai
**Constitutional AI**
**What is Constitutional AI?**
Constitutional AI (CAI) is an alignment approach by Anthropic that uses a set of principles to guide AI behavior, reducing reliance on human feedback for every scenario.
**Core Concept**
Instead of collecting human feedback for every case, define principles (a "constitution") that the model uses for self-improvement.
**The CAI Process**
**Stage 1: Supervised Learning with Self-Critique**
```
1. Generate initial response
2. Critique response against principles
3. Revise response based on critique
4. Fine-tune on revised responses
```
**Stage 2: RLHF with AI Feedback (RLAIF)**
```
1. Generate response pairs
2. AI evaluates which is better (using principles)
3. Train reward model on AI preferences
4. RLHF as usual
```
**Example Constitution Principles**
```
- Be helpful, harmless, and honest
- Refuse to help with illegal activities
- Correct mistakes when pointed out
- Express uncertainty when appropriate
- Avoid stereotypes and bias
- Protect user privacy
- Do not pretend to be human
```
**Self-Critique Example**
```
[Original response]: [potentially harmful content]
[Critique]: This response violates the principle of being harmless
because it provides information that could be used to harm others.
[Revised response]: I cannot provide that information because it
could be used to cause harm. Instead, let me suggest...
```
**Benefits**
| Benefit | Description |
|---------|-------------|
| Scalable | Less human annotation needed |
| Transparent | Principles are explicit |
| Consistent | Same principles applied everywhere |
| Maintainable | Update principles as needed |
**Implementation Approach**
```python
def constitutional_revision(response: str, principles: list) -> str:
# Self-critique
critique = llm.generate(f"""
Given these principles: {principles}
Critique this response:
{response}
Identify any violations of the principles.
""")
# Revision
revised = llm.generate(f"""
Original response: {response}
Critique: {critique}
Generate a revised response that addresses the critique
while remaining helpful.
""")
return revised
```
**Comparison to RLHF**
| Aspect | RLHF | CAI |
|--------|------|-----|
| Human involvement | Every preference | Define principles once |
| Scalability | Limited by humans | Highly scalable |
| Transparency | Implicit in data | Explicit principles |
| Consistency | Varies with annotators | Consistent |
Constitutional AI is foundational to Anthropic Claude models.