Hierarchical RL is a reinforcement learning framework that decomposes complex tasks into a hierarchy of subtasks — a high-level policy selects subtasks (goals, options, or skills), and low-level policies execute them, enabling temporally abstracted decision-making over long horizons.

Hierarchical RL Frameworks

• Options Framework: Define options (macro-actions) with initiation sets, policies, and termination conditions.
• Feudal Networks (FuN): A manager sets goals, a worker executes primitive actions to achieve those goals.
• HAM: Hierarchies of Abstract Machines — constrain the policy space with partial programs.
• MAXQ: Decompose the value function into a hierarchy of subtask values.

Why It Matters

• Long Horizons: Complex tasks require planning over hundreds of steps — hierarchy provides temporal abstraction.
• Transfer: Skills learned for one task transfer to related tasks — modular, reusable components.
• Exploration: High-level exploration over goals is more efficient than low-level random exploration.

Hierarchical RL is divide and conquer for decision-making — decomposing complex tasks into manageable subtasks with multi-level policies.