inverse reinforcement learning irl
# Inverse Reinforcement Learning (IRL)
## Introduction & Motivation
Inverse Reinforcement Learning: infer reward functions from demonstrations. Reward discovery; preference learning. Applications: understanding behavior, transfer learning, explainability.
Motivation: Extract reward signals from expert behavior.
Applications: Reward discovery, behavior understanding, transfer learning.
---
## Core Concepts & Theory
### Maximum Entropy IRL
Find most uncertain reward consistent with demos.
### Deep IRL
Learn reward features via deep learning.
### Guided Cost Learning
Combine demonstrations and constraints.
### Preference Learning
Infer from relative preferences.
---
## Mathematical Formulation
MaxEnt IRL Objective:
$$\max_r H(\pi^*) ext{ subject to } \mathbb{E}[\phi(s,a)]_{\pi^*} = \mathbb{E}[\phi(s,a)]_{ ext{demo}}$$
Reward Function:
$$r(s,a) = w^T \phi(s,a)$$
Expert Likelihood:
$$P( au | r) \propto \exp(- ext{cost}( au))$$
---
## Advanced Theory & Extensions
### Apprenticeship Learning
Learn reward and policy jointly.
### Cooperative IRL
Human-robot collaborative reward learning.
### Hierarchical IRL
Reward structures at multiple levels.
---
## Computational Considerations
Feature computation: O(trajectories·trajectory_len).
Optimization: O(iterations·RL_solver).
Matrix operations: O(features²).
---
## Practical Implementation Strategies
### Feature Engineering
Design reward features carefully.
### Demonstration Preprocessing
Clean and normalize trajectories.
### Validation
Test learned rewards on held-out demos.
---
## Benchmark Datasets & Evaluation
DAPG: Manipulation tasks with demonstrations.
MuJoCo: Continuous control tasks.
Atari: Discrete action environments.
---
## Key Challenges & Limitations
### Non-Identifiability
Multiple rewards explain same behavior.
### Scalability
Expensive optimization.
### Feature Design
Manual feature engineering required.
---
## Hyperparameter Tuning
Learning rate (r): 1e-4 to 1e-2.
Demonstration sample size: 10-100 trajectories.
Feature dimension: 10-1000.
---
## Real-World Applications & Case Studies
Robot Learning: Infer task rewards.
Behavior Understanding: Extract goal functions.
Transfer Learning: Use learned rewards for new tasks.
---
## Integration with Other Methods
Inverse RL + imitation learning for reward-guided behavior; + meta-learning for task inference.
---
## Summary & Key Takeaways
Inverse RL via maximum entropy enables reward inference from expert demonstrations.
Principles:
1. Reward inference: Discover reward functions.
2. Maximum entropy: Least assumptive solution.
3. Feature matching: Distribution alignment.
4. Constraints: Incorporate domain knowledge.
5. Validation: Test on new tasks.
---
---
## Appendix: Practical Labs
### Lab 1: Reward Inference
import numpy as np
def infer_reward(demonstrations, features, learning_rate=0.01, iterations=100):
"""Infer reward weights from demonstrations"""
weights = np.random.randn(features.shape[1]) * 0.01
for _ in range(iterations):
# Compute predicted rewards
predicted_rewards = features @ weights
# Gradient update (simplified)
gradient = features.T @ (predicted_rewards - np.mean(predicted_rewards))
weights -= learning_rate * gradient
return weights
# Test
np.random.seed(42)
demos = np.random.rand(50, 10)
features = np.random.randn(50, 20)
weights = infer_reward(demos, features)
assert weights.shape == (20,), "Correct weight shape"
print("✓ Reward inference working")
if __name__ == "__main__":
print("Lab 1: RewardInference - PASSED")### Lab 2: Feature Matching
import numpy as np
def feature_matching_error(expert_features, learned_features):
"""Compute feature matching error"""
expert_mean = np.mean(expert_features, axis=0)
learned_mean = np.mean(learned_features, axis=0)
error = np.linalg.norm(expert_mean - learned_mean)
return error
# Test
np.random.seed(42)
expert = np.random.rand(50, 20)
learned = expert + np.random.randn(50, 20) * 0.1
error = feature_matching_error(expert, learned)
assert error >= 0, "Error non-negative"
print("✓ Feature matching working")
if __name__ == "__main__":
print("Lab 2: FeatureMatching - PASSED")### Lab 3: Trajectory Reward
import numpy as np
def compute_trajectory_reward(trajectory, reward_weights, features):
"""Compute reward for full trajectory"""
traj_features = features @ trajectory.T
traj_reward = np.sum(traj_features @ reward_weights)
return traj_reward
# Test
np.random.seed(42)
trajectory = np.random.rand(10, 4)
weights = np.random.randn(20)
features = np.random.randn(4, 20)
reward = compute_trajectory_reward(trajectory, weights, features)
assert np.isfinite(reward), "Reward finite"
print("✓ Trajectory reward working")
if __name__ == "__main__":
print("Lab 3: TrajectoryReward - PASSED")### Lab 4: MaxEnt Entropy
import numpy as np
def compute_policy_entropy(policy_probs):
"""Compute policy entropy"""
# Clip for numerical stability
policy_probs = np.clip(policy_probs, 1e-10, 1)
entropy = -np.sum(policy_probs * np.log(policy_probs))
return entropy
# Test
np.random.seed(42)
policy = np.ones(10) / 10
entropy = compute_policy_entropy(policy)
assert entropy > 0, "Entropy positive"
print("✓ Policy entropy working")
if __name__ == "__main__":
print("Lab 4: PolicyEntropy - PASSED")