reinforcement learning hierarchical

**Hierarchical RL** is a **reinforcement learning framework that decomposes complex tasks into a hierarchy of subtasks** — a high-level policy selects subtasks (goals, options, or skills), and low-level policies execute them, enabling temporally abstracted decision-making over long horizons. **Hierarchical RL Frameworks** - **Options Framework**: Define options (macro-actions) with initiation sets, policies, and termination conditions. - **Feudal Networks (FuN)**: A manager sets goals, a worker executes primitive actions to achieve those goals. - **HAM**: Hierarchies of Abstract Machines — constrain the policy space with partial programs. - **MAXQ**: Decompose the value function into a hierarchy of subtask values. **Why It Matters** - **Long Horizons**: Complex tasks require planning over hundreds of steps — hierarchy provides temporal abstraction. - **Transfer**: Skills learned for one task transfer to related tasks — modular, reusable components. - **Exploration**: High-level exploration over goals is more efficient than low-level random exploration. **Hierarchical RL** is **divide and conquer for decision-making** — decomposing complex tasks into manageable subtasks with multi-level policies.