Model-Based Reinforcement Learning
# Model-Based Reinforcement Learning
## Introduction & Motivation
Model-based RL: learn environment model for planning. Combines learned dynamics with action optimization. Improves sample efficiency through imagined rollouts.
Motivation: Reduce environment interactions via learned models.
Applications: Robotics, planning, model-predictive control.
---
## Core Concepts & Theory
### Dynamics Model
Predict next state from state-action.
### Planning
Optimize actions using model.
### Model Uncertainty
Quantify prediction uncertainty.
### Planning Horizon
Look-ahead trajectory length.
---
## Mathematical Formulation
Dynamics Model:
$$s_{t+1} = f(s_t, a_t) + \epsilon$$
Objective (Planning):
$$\max_{\mathbf{a}} \sum_{t=0}^{H} r(s_t, a_t)$$
Model Loss:
$$L = \|s_{t+1} - \hat{s}_{t+1}\|^2$$
---
## Advanced Theory & Extensions
### Ensemble Models
Multiple dynamics models.
### Latent Space Models
Learn in compressed representation.
### Imagination Augmentation
Use imagined rollouts for learning.
---
## Computational Considerations
Model Training: O(D²).
Planning: O(H·A·D) for H horizon.
Total: O(D² + H·A·D).
---
## Practical Implementation Strategies
### Model Initialization
Good initialization critical.
### Planning Optimization
Cross-entropy method or CEM.
### Uncertainty Quantification
Ensemble disagreement.
---
## Benchmark Datasets & Evaluation
MuJoCo: Continuous control.
MBPO: Model-based offline RL.
Robotics: Real-world tasks.
---
## Key Challenges & Limitations
### Model Errors
Compound over horizon.
### Planning Complexity
NP-hard optimization.
### Distribution Shift
Model learns biased dynamics.
---
## Hyperparameter Tuning
Planning horizon: 5-15 steps.
Model ensemble: 5-10 networks.
Uncertainty scale: 0.01-0.1.
---
## Real-World Applications & Case Studies
Robotics: Trajectory optimization.
Control: Model-predictive control.
Planning: Long-horizon reasoning.
---
## Integration with Other Methods
Model-based + model-free; + planning; + learning.
---
## Summary & Key Takeaways
Model-based RL learns environment dynamics for efficient planning.
Principles:
1. Dynamics: Learn environment model.
2. Planning: Optimize with lookahead.
3. Uncertainty: Account for model errors.
4. Ensemble: Multiple predictions.
5. Efficiency: Reduce environment interactions.
---
## Appendix: Practical Labs
### Lab 1: Dynamics Model Training
import numpy as np
def train_dynamics_model(states, actions, next_states, model_weights, iterations=100, lr=0.01):
"""Train neural network dynamics model"""
for _ in range(iterations):
# Batch forward pass
batch_indices = np.random.randint(len(states), size=32)
# Simplified: linear prediction
for idx in batch_indices:
s, a, s_next = states[idx], actions[idx], next_states[idx]
# Predict next state
sa = np.concatenate([s, a])
s_pred = sa @ model_weights
# MSE loss
error = s_next - s_pred
loss = np.sum(error ** 2)
# Update
gradient = -2 * error @ np.concatenate([s, a]).T
model_weights += lr * gradient
return model_weights
states = np.random.randn(100, 10)
actions = np.random.randn(100, 2)
next_states = states + actions.mean(axis=1, keepdims=True) * 0.1
model = np.random.randn(12, 10) * 0.01
model = train_dynamics_model(states, actions, next_states, model)
print(f"✓ Dynamics model trained: shape={model.shape}")### Lab 2: Planning with Learned Model
import numpy as np
def plan_trajectory(model, initial_state, action_dim=2, horizon=10):
"""Plan trajectory using learned model"""
state = initial_state.copy()
trajectory = [state.copy()]
for step in range(horizon):
# Random action (simplified; would use optimization)
action = np.random.randn(action_dim) * 0.5
action = np.clip(action, -1, 1)
# Predict next state
sa = np.concatenate([state, action])
state = sa @ model
trajectory.append(state.copy())
return np.array(trajectory)
model = np.random.randn(12, 10) * 0.01
initial = np.random.randn(10)
traj = plan_trajectory(model, initial, horizon=5)
print(f"✓ Planned trajectory: {len(traj)} steps, shape={traj.shape}")### Lab 3: Model Ensemble
import numpy as np
class DynamicsEnsemble:
def __init__(self, state_dim=10, action_dim=2, num_models=5):
self.state_dim = state_dim
self.action_dim = action_dim
self.num_models = num_models
# Ensemble of dynamics models
self.models = [np.random.randn(state_dim + action_dim, state_dim) * 0.01
for _ in range(num_models)]
def predict_next_state(self, state, action):
"""Predict next state from ensemble"""
predictions = []
for model in self.models:
sa = np.concatenate([state, action])
s_next = sa @ model
predictions.append(s_next)
return np.array(predictions)
def predict_mean_and_std(self, state, action):
"""Get mean and uncertainty"""
predictions = self.predict_next_state(state, action)
mean = np.mean(predictions, axis=0)
std = np.std(predictions, axis=0)
return mean, std
def train(self, states, actions, next_states, iterations=10):
"""Train all models"""
for model_idx, model in enumerate(self.models):
for _ in range(iterations):
idx = np.random.randint(len(states))
sa = np.concatenate([states[idx], actions[idx]])
error = next_states[idx] - sa @ model
model += 0.01 * error @ np.concatenate([states[idx], actions[idx]]).T
ensemble = DynamicsEnsemble()
states = np.random.randn(100, 10)
actions = np.random.randn(100, 2)
next_states = states * 0.9 + np.random.randn(100, 10) * 0.1
ensemble.train(states, actions, next_states)
state = np.random.randn(10)
action = np.random.randn(2)
mean, std = ensemble.predict_mean_and_std(state, action)
print(f"✓ Dynamics ensemble: mean shape={mean.shape}, std shape={std.shape}")### Lab 4: Model-Based Policy Optimization
import numpy as np
class ModelBasedAgent:
def __init__(self, state_dim=10, action_dim=2):
self.state_dim = state_dim
self.action_dim = action_dim
# Dynamics model
self.model = np.random.randn(state_dim + action_dim, state_dim) * 0.01
# Value network
self.value = np.random.randn(state_dim, 1) * 0.01
def plan_actions(self, state, horizon=5, num_samples=100):
"""Plan actions using CEM"""
best_actions = None
best_return = -float('inf')
for _ in range(num_samples):
# Sample action sequence
actions = np.random.randn(horizon, self.action_dim) * 0.5
# Rollout with model
s = state.copy()
trajectory_return = 0
for action in actions:
# Predict next state
sa = np.concatenate([s, action])
s = sa @ self.model
# Compute reward (value-based)
reward = (s @ self.value)[0]
trajectory_return += reward
if trajectory_return > best_return:
best_return = trajectory_return
best_actions = actions.copy()
return best_actions, best_return
def update_model(self, state, action, next_state, learning_rate=0.01):
"""Update dynamics model"""
sa = np.concatenate([state, action])
error = next_state - sa @ self.model
self.model += learning_rate * error @ sa.T
agent = ModelBasedAgent()
state = np.random.randn(10)
actions, ret = agent.plan_actions(state, horizon=3)
print(f"✓ Model-based planning: actions shape={actions.shape}, return={ret:.3f}")---