Inverse Reinforcement Learning
# Inverse Reinforcement Learning
## Introduction & Motivation
IRL: infer reward function from expert demonstrations. Reverse problem of RL. Enables learning from human behavior without explicit reward specification.
Motivation: Understand underlying objectives from observed behavior.
Applications: Imitation learning, autonomous systems, preference learning.
---
## Core Concepts & Theory
### Reward Function Recovery
Infer rewards from trajectories.
### Expert Demonstration
High-quality behavior samples.
### Ambiguity Problem
Multiple rewards explain same behavior.
### Maximum Entropy IRL
Principle for selecting among solutions.
---
## Mathematical Formulation
IRL Problem:
$$\max_R ext{likelihood}( au_{ ext{expert}} | R)$$
Maximum Entropy:
$$R^* = \arg\min_R \| au_{ ext{expert}} - \pi_R^*\|^2$$
Trajectory Distribution:
$$P( au|R) \propto \exp(\sum_t r(s_t, a_t))$$
---
## Advanced Theory & Extensions
### Bayesian IRL
Probabilistic reward inference.
### Deep IRL
Neural network reward functions.
### Generative Adversarial IRL
Adversarial reward learning.
---
## Computational Considerations
Optimization: Nested RL + inference.
Trajectories: O(T·D) per expert trajectory.
Total: O(K·T·D²) for K demonstrations.
---
## Practical Implementation Strategies
### Feature-Based Rewards
Linear reward combinations.
### Constraint Satisfaction
Match expert performance.
### Multiple Demonstrations
Leverage diverse examples.
---
## Benchmark Datasets & Evaluation
MuJoCo: Continuous control.
Gridworlds: Simple environments.
Real Robotics: Learning from humans.
---
## Key Challenges & Limitations
### Non-Uniqueness
Many rewards fit data.
### Computation
Expensive nested optimization.
### Expert Quality
Sensitive to demonstration noise.
---
## Hyperparameter Tuning
Feature scaling: Normalize inputs.
Regularization: L2 on reward weights.
Demonstrations: 5-20 trajectories.
---
## Real-World Applications & Case Studies
Robotics: Learning from human demonstration.
Autonomous Systems: Preference inference.
AI Safety: Understanding human values.
---
## Integration with Other Methods
IRL + imitation learning; + preference learning; + human feedback.
---
## Summary & Key Takeaways
IRL recovers reward functions from expert behavior.
Principles:
1. Inverse: Reward inference from data.
2. Ambiguity: Multiple valid solutions.
3. Maximum Entropy: Principle for selection.
4. Expert: Demonstration quality critical.
5. Features: Represent reward space.
---
## Appendix: Practical Labs
### Lab 1: Linear Reward Function
import numpy as np
def linear_reward(state, action, weights):
"""Compute linear reward from features"""
features = np.concatenate([state, action])
reward = np.dot(weights, features)
return reward
def infer_reward_weights(expert_trajectories, learning_rate=0.01, iterations=100):
"""Infer reward weights from expert demos"""
weights = np.random.randn(12) * 0.1
for _ in range(iterations):
total_loss = 0
for traj in expert_trajectories:
states, actions = traj
# Compute expert performance
expert_return = sum(linear_reward(s, a, weights) for s, a in zip(states, actions))
# Gradient update to maximize expert return
for s, a in zip(states, actions):
features = np.concatenate([s, a])
weights += learning_rate * features * expert_return / len(states)
return weights
# Generate expert trajectories
expert_trajs = [
(np.random.randn(5, 10), np.random.randn(5, 2))
for _ in range(3)
]
weights = infer_reward_weights(expert_trajs)
print(f"✓ Inferred weights: shape={weights.shape}, mean={weights.mean():.3f}")### Lab 2: Maximum Entropy IRL
import numpy as np
def compute_policy_from_reward(reward_func, state, actions):
"""Compute policy via softmax over rewards"""
q_values = np.array([reward_func(state, a) for a in actions])
# Temperature-scaled softmax
q_max = np.max(q_values)
probs = np.exp(q_values - q_max) / np.sum(np.exp(q_values - q_max))
return probs
def max_entropy_loss(expert_trajs, learner_trajs, reward_weights):
"""Maximum entropy IRL loss"""
# Expert likelihood
expert_loss = sum(
sum(np.dot(reward_weights, np.concatenate([s, a])) for s, a in traj)
for traj in expert_trajs
)
# Learner likelihood (minimize)
learner_loss = sum(
sum(np.dot(reward_weights, np.concatenate([s, a])) for s, a in traj)
for traj in learner_trajs
)
# Total: maximize expert - learner
return learner_loss - expert_loss
expert_trajs = [(np.random.randn(3, 10), np.random.randn(3, 2))]
learner_trajs = [(np.random.randn(3, 10), np.random.randn(3, 2))]
weights = np.random.randn(12)
loss = max_entropy_loss(expert_trajs, learner_trajs, weights)
print(f"✓ Max entropy loss: {loss:.3f}")### Lab 3: Feature Extraction
import numpy as np
def extract_features(state, action):
"""Extract features for reward representation"""
features = {
'state_norm': np.linalg.norm(state),
'action_norm': np.linalg.norm(action),
'state_action_dot': np.dot(state[:2], action),
'goal_distance': np.linalg.norm(state[:2]), # Assume first 2 dims are position
}
# Convert to feature vector
feature_vec = np.array(list(features.values()))
return feature_vec
state = np.random.randn(10)
action = np.random.randn(2)
features = extract_features(state, action)
print(f"✓ Extracted features: {len(features)} dimensions")### Lab 4: Complete IRL Agent
import numpy as np
class IRLAgent:
def __init__(self, state_dim=10, action_dim=2, feature_dim=4):
self.state_dim = state_dim
self.action_dim = action_dim
self.feature_dim = feature_dim
# Learned reward weights
self.reward_weights = np.random.randn(feature_dim) * 0.1
def extract_features(self, state, action):
"""Feature extraction"""
s_norm = np.linalg.norm(state) / 10
a_norm = np.linalg.norm(action) / 5
dot = np.dot(state[:2], action) / 10
goal_dist = np.linalg.norm(state[:2]) / 10
return np.array([s_norm, a_norm, dot, goal_dist])
def compute_reward(self, state, action):
"""Compute reward from learned weights"""
features = self.extract_features(state, action)
reward = np.dot(self.reward_weights, features)
return reward
def learn_from_expert(self, expert_trajectories, learning_rate=0.01, iterations=50):
"""Learn reward weights from demonstrations"""
for iteration in range(iterations):
for traj_states, traj_actions in expert_trajectories:
# Compute trajectory reward
traj_reward = sum(
self.compute_reward(s, a)
for s, a in zip(traj_states, traj_actions)
)
# Update weights to maximize expert reward
for s, a in zip(traj_states, traj_actions):
features = self.extract_features(s, a)
self.reward_weights += learning_rate * features * traj_reward / len(traj_states)
agent = IRLAgent()
expert_trajs = [
(np.random.randn(5, 10), np.random.randn(5, 2))
for _ in range(3)
]
agent.learn_from_expert(expert_trajs)
print(f"✓ IRL agent learned: weights shape={agent.reward_weights.shape}")---