Exploration in Reinforcement Learning
# Exploration in Reinforcement Learning
## Introduction & Motivation
Exploration: discovering optimal policies through environmental interaction. Balance exploitation with discovery. Critical for long-horizon and sparse reward problems.
Motivation: Enable efficient discovery of rewarding behaviors.
Applications: Complex environments, sparse rewards, long horizons.
---
## Core Concepts & Theory
### Exploration-Exploitation Tradeoff
Balancing discovery and performance.
### Intrinsic Motivation
Curiosity-driven exploration.
### Uncertainty Estimation
Confidence in value estimates.
### Empowerment
Maximize future action space.
---
## Mathematical Formulation
ε-Greedy:
$$a = \begin{cases} a^* & ext{w.p. } 1-\epsilon \\ ext{random} & ext{w.p. } \epsilon \end{cases}$$
Upper Confidence Bound:
$$a^* = \arg\max_a (Q(a) + c \sqrt{\frac{\ln t}{N(a)}})$$
Curiosity Reward:
$$r_{ ext{curiosity}} = \|\hat{s}_{t+1} - s_{t+1}\|^2$$
---
## Advanced Theory & Extensions
### Count-Based Exploration
Visit count regularization.
### Random Network Distillation
Prediction error as novelty.
### Empowerment-Based
Maximize mutual information.
---
## Computational Considerations
ε-Greedy: O(1).
UCB: O(log t).
Curiosity: O(D²) for prediction network.
---
## Practical Implementation Strategies
### Epsilon Decay
Decrease exploration over time.
### Bonus Rewards
Augment environment rewards.
### Ensemble Uncertainty
Use network variance.
---
## Benchmark Datasets & Evaluation
Atari: Sparse reward exploration.
MuJoCo: Continuous exploration.
Goal Environments: Exploration metrics.
---
## Key Challenges & Limitations
### Exploration Noise
Trade-off with exploitation.
### Novelty vs Progress
Distinguish between.
### Scalability
Expensive for high-dimensional spaces.
---
## Hyperparameter Tuning
ε decay: 0.995-0.999.
Curiosity scale: 0.01-0.1.
UCB c: 1.0-2.0.
---
## Real-World Applications & Case Studies
Robotics: Safe exploration.
Game AI: Complex strategy learning.
Optimization: Multi-objective search.
---
## Integration with Other Methods
Exploration + uncertainty; + hierarchical RL; + meta-learning.
---
## Summary & Key Takeaways
Exploration enables efficient discovery of optimal policies.
Principles:
1. Tradeoff: Exploit vs explore.
2. Uncertainty: Reduce epistemic uncertainty.
3. Intrinsic: Curiosity-driven learning.
4. Empowerment: Maximize capabilities.
5. Novelty: Discover new states.
---
## Appendix: Practical Labs
### Lab 1: Epsilon-Greedy Exploration
import numpy as np
def epsilon_greedy_action(Q, state, epsilon=0.1):
"""Select action with ε-greedy policy"""
if np.random.rand() < epsilon:
return np.random.randint(Q.shape[1])
else:
return np.argmax(Q[state])
def epsilon_decay_schedule(initial_epsilon=1.0, decay=0.995, steps=10000):
"""Epsilon decay over training"""
epsilons = []
eps = initial_epsilon
for step in range(steps):
epsilons.append(eps)
eps *= decay
return epsilons
Q = np.random.randn(5, 4)
actions = [epsilon_greedy_action(Q, 0, epsilon=0.1) for _ in range(100)]
epsilons = epsilon_decay_schedule(steps=1000)
print(f"✓ ε-greedy: {len(actions)} actions selected")
print(f"✓ ε-decay: initial={epsilons[0]:.3f}, final={epsilons[-1]:.4f}")### Lab 2: Curiosity-Driven Exploration
import numpy as np
def compute_curiosity_reward(predicted_next_state, actual_next_state):
"""Compute curiosity as prediction error"""
prediction_error = np.linalg.norm(predicted_next_state - actual_next_state) ** 2
return prediction_error
def curiosity_bonus(env_reward, curiosity_reward, curiosity_scale=0.01):
"""Combine environment and curiosity rewards"""
total_reward = env_reward + curiosity_scale * curiosity_reward
return total_reward
# Simulation
env_reward = 1.0
predicted = np.random.randn(10)
actual = predicted + np.random.randn(10) * 0.5
curiosity = compute_curiosity_reward(predicted, actual)
total_reward = curiosity_bonus(env_reward, curiosity, curiosity_scale=0.05)
print(f"✓ Curiosity: prediction error={curiosity:.3f}")
print(f"✓ Total reward: {env_reward} + {curiosity*0.05:.3f} = {total_reward:.3f}")### Lab 3: Upper Confidence Bound
import numpy as np
def ucb_action(Q_values, counts, c=1.0, t=100):
"""UCB action selection"""
# Compute confidence bounds
ucb_scores = Q_values + c * np.sqrt(np.log(t) / (counts + 1))
action = np.argmax(ucb_scores)
return action
def ucb_exploration_comparison(num_actions=4, steps=1000):
"""Compare UCB vs ε-greedy"""
Q_ucb = np.zeros(num_actions)
Q_eps = np.zeros(num_actions)
counts_ucb = np.zeros(num_actions)
counts_eps = np.zeros(num_actions)
for t in range(steps):
# UCB
action_ucb = ucb_action(Q_ucb, counts_ucb, c=1.4, t=t+1)
counts_ucb[action_ucb] += 1
# ε-greedy
epsilon = 1.0 / (1.0 + t/100) # Decay
if np.random.rand() < epsilon:
action_eps = np.random.randint(num_actions)
else:
action_eps = np.argmax(Q_eps)
counts_eps[action_eps] += 1
return counts_ucb, counts_eps
counts_ucb, counts_eps = ucb_exploration_comparison()
print(f"✓ UCB visits: {counts_ucb}")
print(f"✓ ε-greedy visits: {counts_eps}")### Lab 4: Exploration with Uncertainty
import numpy as np
class ExplorationAgent:
def __init__(self, state_dim=10, action_dim=4, num_networks=5):
self.state_dim = state_dim
self.action_dim = action_dim
self.num_networks = num_networks
# Ensemble of Q-networks
self.Q_ensemble = [np.random.randn(state_dim, action_dim) * 0.01 for _ in range(num_networks)]
def compute_uncertainty(self, state):
"""Estimate uncertainty from ensemble"""
q_values = [Q[state] for Q in self.Q_ensemble]
q_values = np.array(q_values)
# Disagreement among networks
uncertainty = np.std(q_values, axis=0)
return uncertainty
def select_action_ucb(self, state, c=1.0):
"""UCB action selection using ensemble"""
# Mean Q-values
q_mean = np.mean([Q[state] for Q in self.Q_ensemble], axis=0)
# Uncertainty bonus
uncertainty = self.compute_uncertainty(state)
# UCB
ucb_scores = q_mean + c * uncertainty
action = np.argmax(ucb_scores)
return action
def update(self, state, action, reward, next_state):
"""Update all ensemble networks"""
for i, Q in enumerate(self.Q_ensemble):
q_next = np.max(Q[next_state])
q_target = reward + 0.99 * q_next
td_error = q_target - Q[state, action]
Q[state, action] += 0.01 * td_error
agent = ExplorationAgent()
state = 0
action = agent.select_action_ucb(state)
print(f"✓ Ensemble exploration agent: action={action}")---