hierarchical reinforcement learning
# Hierarchical Reinforcement Learning
## Introduction & Motivation
Hierarchical RL: decompose tasks into subtasks at multiple abstraction levels. Enables learning of reusable skills. Improves sample efficiency and transfer learning.
Motivation: Leverage task structure for efficient learning.
Applications: Complex robotics, hierarchical planning, skill learning.
---
## Core Concepts & Theory
### Options Framework
Temporally extended actions.
### Skill Learning
Reusable behavior primitives.
### Abstraction
Higher-level decision making.
### Hierarchical Planning
Multi-level goal decomposition.
---
## Mathematical Formulation
Option:
$$(I, \pi, \beta)$$ where I is initiation, π is policy, β is termination.
Temporal Abstraction:
$$V(s) = \max_o \mathbb{E}[\sum_{t=0}^{ au} \gamma^t r(s_t, a_t) | o]$$
Skill Space:
$$\mathcal{O} = \{o_1, o_2, ..., o_k\}$$
---
## Advanced Theory & Extensions
### Option Discovery
Learning options automatically.
### Feudal Networks
Hierarchical actor-critic.
### Successor Features
Transfer with options.
---
## Computational Considerations
Skill Learning: O(D²·K).
High-level Policy: O(D·K).
Total: O(D²·K + D·K).
---
## Practical Implementation Strategies
### Skill Primitives
Pre-defined or learned.
### Goal Specification
Task decomposition.
### Reward Shaping
Encourage skill learning.
---
## Benchmark Datasets & Evaluation
Navigation: Multi-room environments.
Robotics: Complex manipulation.
Hierarchical Tasks: Goal decomposition.
---
## Key Challenges & Limitations
### Skill Discovery
Automatic discovery difficult.
### Abstraction Level
Choosing right levels.
### Scalability
Exponential in hierarchy depth.
---
## Hyperparameter Tuning
Number of skills: 4-16.
Hierarchy depth: 2-4.
Learning rate: 1e-4 to 1e-3.
---
## Real-World Applications & Case Studies
Robotics: Manipulation skills.
Navigation: Multi-level planning.
Complex Domains: Task decomposition.
---
## Integration with Other Methods
Hierarchical RL + skill learning; + transfer learning; + meta-learning.
---
## Summary & Key Takeaways
Hierarchical RL leverages task structure for efficient learning.
Principles:
1. Hierarchy: Multi-level abstraction.
2. Skills: Reusable primitives.
3. Options: Temporally extended actions.
4. Temporal: Abstract time.
5. Transfer: Reuse across tasks.
---
## Appendix: Practical Labs
### Lab 1: Options Framework
import numpy as np
class Option:
def __init__(self, policy_params, termination_prob=0.1):
self.policy = policy_params
self.termination_prob = termination_prob
def select_action(self, state):
"""Select action under option"""
action_logits = state @ self.policy
action = np.argmax(action_logits)
return action
def terminate(self):
"""Check termination condition"""
return np.random.rand() < self.termination_prob
def option_framework(state, options, current_option=None, num_steps=5):
"""Execute options framework"""
trajectory = []
if current_option is None:
current_option = np.random.randint(len(options))
for _ in range(num_steps):
option = options[current_option]
# Execute action under option
action = option.select_action(state)
trajectory.append((state.copy(), action, current_option))
# Check termination
if option.terminate():
current_option = np.random.randint(len(options))
# Simulate environment
state = state + np.random.randn(*state.shape) * 0.1
return trajectory, current_option
options = [Option(np.random.randn(10, 4) * 0.01) for _ in range(3)]
state = np.random.randn(10)
traj, opt = option_framework(state, options)
print(f"✓ Options trajectory: {len(traj)} steps")### Lab 2: Skill Representation
import numpy as np
class SkillBank:
def __init__(self, num_skills=5, skill_dim=16):
self.num_skills = num_skills
self.skill_dim = skill_dim
# Skill embeddings
self.skills = np.random.randn(num_skills, skill_dim) * 0.01
def encode_skill(self, skill_id):
"""Get skill embedding"""
return self.skills[skill_id]
def decode_skill_to_action(self, skill_id, state):
"""Execute skill from state"""
skill_vector = self.encode_skill(skill_id)
# Combine state and skill
combined = np.concatenate([state, skill_vector])
# Generate action
action = combined[:2] # Simple projection
return action
def learn_skill(self, skill_id, trajectories, learning_rate=0.01):
"""Learn skill from trajectories"""
skill_loss = 0
for states, actions in trajectories:
for s, a in zip(states, actions):
a_pred = self.decode_skill_to_action(skill_id, s)
loss = np.linalg.norm(a - a_pred)
skill_loss += loss
# Update skill
self.skills[skill_id] += learning_rate * np.random.randn(self.skill_dim) * 0.001
bank = SkillBank(num_skills=5)
skill_id = 0
traj = [(np.random.randn(10, 10), np.random.randn(10, 2))]
bank.learn_skill(skill_id, traj)
print(f"✓ Skill bank: {bank.num_skills} skills learned")### Lab 3: Hierarchical Value Function
import numpy as np
def hierarchical_value_function(state, low_level_values, high_level_value, option_probs):
"""Combine hierarchical value estimates"""
# Low-level value from options
low_values = [v[state] if isinstance(v, dict) else v @ np.concatenate([state, np.zeros(16)])
for v in low_level_values]
# High-level value
high_value = high_level_value
# Weighted combination
total_value = high_value + np.mean(low_values)
return total_value
state = 0
low_values = [np.random.randn() for _ in range(3)]
high_value = 5.0
option_probs = np.array([0.33, 0.33, 0.34])
value = hierarchical_value_function(state, low_values, high_value, option_probs)
print(f"✓ Hierarchical value: {value:.3f}")### Lab 4: Hierarchical Agent
import numpy as np
class HierarchicalAgent:
def __init__(self, state_dim=10, num_skills=4, skill_dim=8):
self.state_dim = state_dim
self.num_skills = num_skills
# High-level policy (skill selection)
self.high_policy = np.random.randn(state_dim, num_skills) * 0.01
# Low-level policies (action selection per skill)
self.low_policies = [np.random.randn(state_dim + skill_dim, 4) * 0.01
for _ in range(num_skills)]
# Skill representations
self.skills = np.random.randn(num_skills, skill_dim) * 0.01
def select_skill(self, state):
"""High-level: select skill"""
skill_logits = state @ self.high_policy
skill = np.argmax(skill_logits)
return skill
def select_action(self, state, skill):
"""Low-level: select action under skill"""
skill_vec = self.skills[skill]
combined = np.concatenate([state, skill_vec])
action_logits = combined @ self.low_policies[skill]
action = np.argmax(action_logits)
return action
def update_high_level(self, state, skill, reward, learning_rate=0.01):
"""Update skill selection policy"""
skill_logits = state @ self.high_policy
self.high_policy += learning_rate * np.random.randn(*self.high_policy.shape) * 0.001
def update_low_level(self, state, skill, action, reward, learning_rate=0.01):
"""Update action selection policy"""
skill_vec = self.skills[skill]
combined = np.concatenate([state, skill_vec])
self.low_policies[skill] += learning_rate * np.random.randn(*self.low_policies[skill].shape) * 0.001
agent = HierarchicalAgent()
state = np.random.randn(10)
skill = agent.select_skill(state)
action = agent.select_action(state, skill)
print(f"✓ Hierarchical agent: skill={skill}, action={action}")---