Imitation Learning Learning from Demonstrations

# Imitation Learning & Learning from Demonstrations

## Introduction & Motivation

Imitation Learning: learn from expert demonstrations. Behavioral cloning, inverse reinforcement learning. Applications: robotics, autonomous driving, complex task learning.

Motivation: Learn tasks from examples rather than reward engineering.

Applications: Robot control, autonomous driving, game playing.

---

## Core Concepts & Theory

### Behavioral Cloning

Supervised learning on expert trajectories.

### Dataset Aggregation (DAgger)

Collect more data in training loop.

### Inverse Reinforcement Learning

Infer reward function from demonstrations.

### Generative Adversarial Imitation Learning (GAIL)

Adversarial framework for imitation.

---

## Mathematical Formulation

Behavioral Cloning Loss:
$$L = \sum_t \|a_t - \pi(s_t)\|^2$$

DAgger Objective:
$$L = \alpha L_{ ext{BC}} + (1-\alpha) L_{ ext{distill}}$$

GAIL Discriminator:
$$L_D = -\mathbb{E}[\log D(s,a)] - \mathbb{E}[\log(1-D(s',a'))]$$

---

## Advanced Theory & Extensions

### One-Shot Imitation Learning

Learn from single demonstrations.

### Meta-Imitation Learning

Few-shot learning from demonstrations.

### Offline Imitation Learning

Learn from batch demonstrations.

---

## Computational Considerations

BC: O(trajectories·trajectory_length·model).

DAgger: O(iterations·expert_queries).

GAIL: O(discriminator_steps·policy_steps).

---

## Practical Implementation Strategies

### Expert Data Collection

Carefully gather demonstrations.

### Action Distribution Matching

Match expert behavior distribution.

### Demonstration Augmentation

Expand limited data.

---

## Benchmark Datasets & Evaluation

DAPG Dataset: Robotic manipulation.

Atari Learning Environment: Video game demonstrations.

CARLA: Autonomous driving simulations.

---

## Key Challenges & Limitations

### Distribution Mismatch

Expert vs. student behavior divergence.

### Limited Demonstrations

Small expert datasets.

### Complex Behaviors

Multi-modal action distributions.

---

## Hyperparameter Tuning

Learning rate: 1e-4 to 1e-3.

Batch size: 32-256.

DAgger mixing ratio (α): 0.5-0.9.

---

## Real-World Applications & Case Studies

Robot Manipulation: Learning from kinesthetic teaching.

Autonomous Driving: Learning from human drivers.

Game Playing: Learning from expert gamers.

---

## Integration with Other Methods

Imitation learning + reinforcement learning for efficient exploration; + inverse RL for reward discovery.

---

## Summary & Key Takeaways

Imitation Learning via behavioral cloning and GAIL enables efficient learning from demonstrations.

Principles:
1. Behavioral cloning: Supervised learning on expert actions.
2. Distribution shift: Training-test mismatch.
3. DAgger: Interactive data collection.
4. GAIL: Adversarial framework.
5. Inverse RL: Reward inference.

---

---

## Appendix: Practical Labs

### Lab 1: Behavioral Cloning

import numpy as np

def behavioral_cloning_loss(policy_actions, expert_actions):
 """Compute behavioral cloning loss"""
 loss = np.mean((policy_actions - expert_actions) ** 2)
 return loss

# Test
np.random.seed(42)
policy = np.random.rand(100, 4)
expert = policy + np.random.randn(100, 4) * 0.1

loss = behavioral_cloning_loss(policy, expert)

assert loss >= 0, "Loss non-negative"
print("✓ Behavioral cloning working")

if __name__ == "__main__":
 print("Lab 1: BehavioralCloning - PASSED")

### Lab 2: DAgger

import numpy as np

def dagger_update(current_policy, expert_policy, alpha=0.5):
 """DAgger mixture update"""
 mixed_policy = alpha * expert_policy + (1 - alpha) * current_policy
 return mixed_policy

# Test
np.random.seed(42)
current = np.random.rand(100, 4)
expert = np.random.rand(100, 4)

mixed = dagger_update(current, expert, alpha=0.7)

assert mixed.shape == current.shape, "Shape preserved"
print("✓ DAgger working")

if __name__ == "__main__":
 print("Lab 2: DAgger - PASSED")

### Lab 3: Trajectory Matching

import numpy as np

def trajectory_distance(traj1, traj2):
 """Compute distance between trajectories"""
 distance = np.mean(np.linalg.norm(traj1 - traj2, axis=1))
 return distance

# Test
np.random.seed(42)
traj1 = np.random.rand(50, 4)
traj2 = traj1 + np.random.randn(50, 4) * 0.1

dist = trajectory_distance(traj1, traj2)

assert dist >= 0, "Distance non-negative"
print("✓ Trajectory matching working")

if __name__ == "__main__":
 print("Lab 3: TrajectoryMatching - PASSED")

### Lab 4: Expert Data Augmentation

import numpy as np

def augment_demonstrations(trajectories, num_augmentations=2, noise_std=0.05):
 """Augment expert demonstrations with noise"""
 augmented = []
 
 for traj in trajectories:
 augmented.append(traj)
 for _ in range(num_augmentations):
 noise = np.random.randn(*traj.shape) * noise_std
 augmented.append(traj + noise)
 
 return augmented

# Test
np.random.seed(42)
trajs = [np.random.rand(20, 4) for _ in range(5)]

augmented = augment_demonstrations(trajs, num_augmentations=2)

assert len(augmented) == 15, "Correct augmentation count"
print("✓ Demonstration augmentation working")

if __name__ == "__main__":
 print("Lab 4: DemonstrationAugmentation - PASSED")