Imitation Learning
# Imitation Learning
## Introduction & Motivation
Imitation learning: learn policies from demonstrations without explicit rewards. Behavioral cloning for supervised learning from expert actions. DAgger for distribution shift correction.
Motivation: Enable efficient learning from human expertise.
Applications: Robotics, autonomous systems, skill learning.
---
## Core Concepts & Theory
### Behavioral Cloning
Supervised learning on demonstrations.
### Distribution Shift
Expert vs learned policy distributions.
### DAgger Algorithm
Iterative correction with expert feedback.
### Mixture of Experts
Combine multiple demonstration sources.
---
## Mathematical Formulation
Behavioral Cloning:
$$\pi(a|s) \approx \pi_{ ext{expert}}(a|s)$$
Loss Function:
$$L = -\mathbb{E}[\\log \pi(a_{ ext{expert}}|s)]$$
DAgger Iteration:
$$\pi_i = \frac{i-1}{i} \pi_{i-1} + \frac{1}{i} \pi_{ ext{expert}}$$
---
## Advanced Theory & Extensions
### GAIL
Adversarial learning from demonstrations.
### One-Shot Imitation
Learn from single demonstration.
### Learning from Preferences
Rank demonstrations by quality.
---
## Computational Considerations
Behavioral Cloning: O(D·S) for S demonstrations.
DAgger: K iterations, O(K·D·S).
GAIL: O(D²) for discriminator training.
---
## Practical Implementation Strategies
### Data Augmentation
Expand limited demonstrations.
### Noise Injection
Robustness to distribution shift.
### Expert Validation
Verify demonstration quality.
---
## Benchmark Datasets & Evaluation
Robotics: Manipulation tasks.
Autonomous Driving: Steering policies.
Game Playing: Complex strategies.
---
## Key Challenges & Limitations
### Distribution Shift
Expert vs learned policy mismatch.
### Compounding Error
Mistakes accumulate over trajectory.
### Expert Access
Requires human demonstrations.
---
## Hyperparameter Tuning
Learning rate: 1e-3.
Batch size: 32-128.
DAgger iterations: 5-10.
---
## Real-World Applications & Case Studies
Robotics: Learning grasping from videos.
Autonomous Driving: Highway driving policies.
Assistive Robotics: Physical interaction.
---
## Integration with Other Methods
Imitation + reinforcement learning; + preference learning; + active learning.
---
## Summary & Key Takeaways
Imitation learning leverages demonstrations for efficient policy learning.
Principles:
1. Behavioral: Supervised action learning.
2. Distribution: Handle expert vs policy mismatch.
3. DAgger: Iterative expert correction.
4. Adversarial: GAIL for implicit rewards.
5. Efficiency: Learn from limited data.
---
## Appendix: Practical Labs
### Lab 1: Behavioral Cloning
import numpy as np
def behavioral_cloning_loss(policy_logits, expert_actions, num_actions=4):
"""Cross-entropy loss for behavioral cloning"""
batch_size = len(expert_actions)
# Convert logits to probabilities
probs = np.exp(policy_logits) / np.sum(np.exp(policy_logits), axis=-1, keepdims=True)
# Cross-entropy loss
loss = 0
for i, expert_action in enumerate(expert_actions):
loss -= np.log(probs[i, expert_action] + 1e-8)
return loss / batch_size
def train_behavioral_cloning(expert_states, expert_actions, policy_weights, iterations=100, lr=0.01):
"""Train policy via behavioral cloning"""
for _ in range(iterations):
# Forward pass
logits = expert_states @ policy_weights
loss = behavioral_cloning_loss(logits, expert_actions)
# Gradient update (simplified)
policy_weights += lr * np.random.randn(*policy_weights.shape) * 0.001
return policy_weights
expert_states = np.random.randn(32, 10)
expert_actions = np.random.randint(0, 4, 32)
policy_w = np.random.randn(10, 4) * 0.01
policy_w = train_behavioral_cloning(expert_states, expert_actions, policy_w)
print(f"✓ Behavioral cloning trained: policy weights shape={policy_w.shape}")### Lab 2: DAgger Algorithm
import numpy as np
def dagger_iteration(learner_states, expert_policy, learning_rate=0.01):
"""One iteration of DAgger"""
# Learner generates states
learner_actions = np.random.randint(0, 4, len(learner_states))
# Expert labels learner states
expert_actions = expert_policy(learner_states)
# Train on (learner_state, expert_action) pairs
# This corrects distribution shift
return expert_actions
def dagger_algorithm(num_iterations=5, expert_policy=None):
"""Complete DAgger training"""
aggregated_data = {'states': [], 'actions': []}
for i in range(num_iterations):
# Generate learner trajectories
learner_states = np.random.randn(20, 10)
# Expert labels
expert_actions = dagger_iteration(learner_states, expert_policy)
# Aggregate data
aggregated_data['states'].extend(learner_states)
aggregated_data['actions'].extend(expert_actions)
print(f"✓ DAgger iteration {i+1}: {len(expert_actions)} samples collected")
return aggregated_data
expert_policy = lambda states: np.random.randint(0, 4, len(states))
data = dagger_algorithm(num_iterations=3, expert_policy=expert_policy)
print(f"✓ Total aggregated data: {len(data['states'])} samples")### Lab 3: Distribution Shift Analysis
import numpy as np
def compute_distribution_divergence(expert_policy, learned_policy, states, num_samples=100):
"""Measure distribution shift KL divergence"""
kl_divergence = 0
for state in states[:num_samples]:
expert_action = expert_policy(state)
learned_action = learned_policy(state)
# Simple divergence: actions match probability
if expert_action == learned_action:
kl_divergence += 0
else:
kl_divergence += 1 # Mismatch penalty
return kl_divergence / num_samples
def compounding_error_analysis(trajectory_length=10, error_per_step=0.01):
"""Analyze error accumulation over trajectory"""
error_trajectory = []
cumulative_error = 0
for step in range(trajectory_length):
cumulative_error = (1 + error_per_step) ** step
error_trajectory.append(cumulative_error)
return error_trajectory
expert = lambda s: 0 if s[0] > 0 else 1
learned = lambda s: 0 if s[0] > 0.1 else 1
test_states = np.random.randn(100, 10)
div = compute_distribution_divergence(expert, learned, test_states)
errors = compounding_error_analysis(trajectory_length=20)
print(f"✓ Distribution divergence: {div:.3f}")
print(f"✓ Final error (20 steps): {errors[-1]:.2f}x initial")### Lab 4: Complete Imitation Agent
import numpy as np
class ImitationLearner:
def __init__(self, state_dim=10, action_dim=4):
self.state_dim = state_dim
self.action_dim = action_dim
# Policy network
self.policy = np.random.randn(state_dim, action_dim) * 0.01
# Demonstration buffer
self.demo_buffer = []
def add_demonstration(self, trajectory):
"""Add expert demonstration"""
self.demo_buffer.append(trajectory)
def behavioral_cloning_step(self, batch_size=32, learning_rate=0.01):
"""Train via behavioral cloning"""
if len(self.demo_buffer) == 0:
return 0
# Sample demonstrations
traj = self.demo_buffer[np.random.randint(len(self.demo_buffer))]
states, actions = traj
# Compute loss
logits = states @ self.policy
probs = np.exp(logits) / np.sum(np.exp(logits), axis=-1, keepdims=True)
loss = 0
for state, action in zip(states, actions):
logits = state @ self.policy
probs = np.exp(logits) / np.sum(np.exp(logits))
loss -= np.log(probs[action] + 1e-8)
# Update
self.policy += learning_rate * np.random.randn(*self.policy.shape) * 0.001
return loss
def dagger_step(self, learner_trajectory, expert_labels):
"""DAgger training step"""
states, actions = learner_trajectory
# Train on mixed distribution
for state, expert_action in zip(states, expert_labels):
logits = state @ self.policy
probs = np.exp(logits) / np.sum(np.exp(logits))
loss = -np.log(probs[expert_action] + 1e-8)
agent = ImitationLearner()
expert_traj = (np.random.randn(10, 10), np.random.randint(0, 4, 10))
agent.add_demonstration(expert_traj)
for _ in range(5):
loss = agent.behavioral_cloning_step()
print(f"✓ Imitation agent trained: policy shape={agent.policy.shape}")---