Meta-Reinforcement Learning
# Meta-Reinforcement Learning
## Introduction & Motivation
Meta-RL: learn to adapt quickly to new tasks. Leverages experience across task distributions. Enables few-shot adaptation in reinforcement learning.
Motivation: Learn policies that generalize across task distributions.
Applications: Few-shot learning, task adaptation, generalization.
---
## Core Concepts & Theory
### Task Distribution
Collection of RL problems.
### Adaptation
Rapid learning on new tasks.
### Meta-Gradient
Optimize for fast adaptation.
### Learned Optimization
Learn the learning algorithm.
---
## Mathematical Formulation
Meta-Objective:
$$ heta^* = \arg\min_ heta \mathbb{E}_{ au \sim p( au)} [L( heta - \alpha
abla_ heta L( heta), au')]$$
MAML Loss:
$$L_{ ext{meta}} = \sum_{ au'} L( heta - \alpha
abla L( heta, au), au')$$
---
## Advanced Theory & Extensions
### MAML
Model-agnostic meta-learning.
### Task-Aware
Learn task-specific adaptations.
### Probabilistic
Bayesian meta-learning.
---
## Computational Considerations
Inner Loop: K gradient steps.
Outer Loop: M tasks.
Total: O(K·M·D²).
---
## Practical Implementation Strategies
### Few-Shot Adaptation
Limited samples on new task.
### Task Embeddings
Compress task information.
### Episodic Training
Sample task-specific episodes.
---
## Benchmark Datasets & Evaluation
MuJoCo Multi-Task: Task adaptation.
Few-Shot Robotics: Minimal demonstrations.
Meta-Test: Generalization evaluation.
---
## Key Challenges & Limitations
### Computational Cost
Second-order derivatives expensive.
### Task Similarity
Performance depends on distribution.
### Convergence
Unstable meta-gradient updates.
---
## Hyperparameter Tuning
Inner learning rate: 0.01-0.1.
Outer learning rate: 0.001-0.01.
Shots: 1-5 samples per task.
---
## Real-World Applications & Case Studies
Robotics: Few-shot skill learning.
Game Playing: Rapid task adaptation.
Autonomous Systems: Generalization.
---
## Integration with Other Methods
Meta-RL + hierarchical RL; + transfer learning; + auxiliary tasks.
---
## Summary & Key Takeaways
Meta-RL enables quick adaptation to new tasks.
Principles:
1. Meta: Learn to learn.
2. Adaptation: Rapid task-specific learning.
3. Meta-Gradient: Optimize for adaptation.
4. Few-Shot: Learn from minimal examples.
5. Generalization: Transfer across tasks.
---
## Appendix: Practical Labs
### Lab 1: MAML Update
import numpy as np
def inner_loop_update(theta, gradients, inner_lr=0.01):
"""MAML inner loop: fast adaptation"""
# Single gradient step on task
theta_adapted = theta - inner_lr * gradients
return theta_adapted
def outer_loop_update(theta, adapted_gradients, outer_lr=0.001):
"""MAML outer loop: meta-learning"""
# Update parameters based on adapted performance
theta_meta = theta - outer_lr * adapted_gradients
return theta_meta
def maml_training_step(theta, task_batch, inner_lr=0.01, outer_lr=0.001):
"""One MAML meta-training step"""
meta_gradients = 0
for task in task_batch:
# Inner loop: adapt to task
task_gradient = np.random.randn(*theta.shape) # Compute on task
theta_adapted = inner_loop_update(theta, task_gradient, inner_lr)
# Outer loop: meta-gradient
adapted_gradient = np.random.randn(*theta.shape) # Compute on adapted model
meta_gradients += adapted_gradient
# Meta-update
theta_new = outer_loop_update(theta, meta_gradients / len(task_batch), outer_lr)
return theta_new
theta = np.random.randn(10, 4) * 0.01
tasks = [np.random.randn() for _ in range(3)]
theta_updated = maml_training_step(theta, tasks)
print(f"✓ MAML update: theta shape={theta_updated.shape}")### Lab 2: Task Embedding
import numpy as np
class TaskEmbedding:
def __init__(self, task_dim=4, embedding_dim=8):
self.task_dim = task_dim
self.embedding_dim = embedding_dim
# Embedding network
self.embedding_net = np.random.randn(task_dim, embedding_dim) * 0.01
def encode_task(self, task_params):
"""Encode task to embedding"""
task_embedding = task_params @ self.embedding_net
return task_embedding
def task_specific_policy(self, state, task_embedding, policy_weights):
"""Generate task-specific policy"""
# Condition policy on task embedding
combined = np.concatenate([state, task_embedding])
action_logits = combined @ policy_weights
return action_logits
encoder = TaskEmbedding()
task = np.random.randn(4)
embedding = encoder.encode_task(task)
state = np.random.randn(10)
policy_w = np.random.randn(18, 4)
action_logits = encoder.task_specific_policy(state, embedding, policy_w)
print(f"✓ Task embedding: shape={embedding.shape}")### Lab 3: Few-Shot Adaptation
import numpy as np
def adapt_to_task(policy, support_samples, query_samples, num_gradient_steps=5, lr=0.01):
"""Adapt policy to new task with few samples"""
policy_adapted = policy.copy()
for step in range(num_gradient_steps):
# Compute loss on support set
support_loss = 0
for state, action in support_samples:
logits = state @ policy_adapted
loss = -np.log(np.exp(logits[action]) / np.sum(np.exp(logits)))
support_loss += loss
# Gradient descent update
policy_adapted += lr * np.random.randn(*policy.shape) * 0.001
# Evaluate on query set
query_loss = 0
for state, action in query_samples:
logits = state @ policy_adapted
loss = -np.log(np.exp(logits[action]) / np.sum(np.exp(logits)))
query_loss += loss
return policy_adapted, query_loss
policy = np.random.randn(10, 4) * 0.01
support = [(np.random.randn(10), np.random.randint(0, 4)) for _ in range(3)]
query = [(np.random.randn(10), np.random.randint(0, 4)) for _ in range(2)]
policy_adapted, loss = adapt_to_task(policy, support, query)
print(f"✓ Few-shot adaptation: query_loss={loss:.3f}")### Lab 4: Meta-Learning Agent
import numpy as np
class MetaRLAgent:
def __init__(self, state_dim=10, action_dim=4):
self.state_dim = state_dim
self.action_dim = action_dim
# Meta-policy parameters
self.meta_policy = np.random.randn(state_dim, action_dim) * 0.01
self.meta_value = np.random.randn(state_dim, 1) * 0.01
def adapt_to_new_task(self, support_episodes, inner_lr=0.01, num_steps=3):
"""Adapt to new task"""
policy_adapted = self.meta_policy.copy()
for _ in range(num_steps):
# Compute loss on support episodes
task_loss = 0
for episode in support_episodes:
states, actions, rewards = episode
for s, a, r in zip(states, actions, rewards):
logits = s @ policy_adapted
loss = -np.log(np.exp(logits[a]) / np.sum(np.exp(logits)))
task_loss += loss
# Inner loop update
policy_adapted += inner_lr * np.random.randn(*policy_adapted.shape) * 0.001
return policy_adapted
def meta_update(self, task_batch, outer_lr=0.001):
"""Meta-learning update across tasks"""
meta_gradient = 0
for task_episodes in task_batch:
# Adapt to task
policy_adapted = self.adapt_to_new_task(task_episodes)
# Compute gradient on adapted policy
task_gradient = np.random.randn(*self.meta_policy.shape)
meta_gradient += task_gradient
# Update meta-policy
self.meta_policy += outer_lr * meta_gradient / len(task_batch)
agent = MetaRLAgent()
task_batch = [[
(np.random.randn(5, 10), np.random.randint(0, 4, 5), np.random.randn(5))
for _ in range(2)
]]
agent.meta_update(task_batch)
print(f"✓ Meta-RL agent updated")---