maxq

**MAXQ** is a **hierarchical RL value decomposition method that breaks down the overall value function into a sum of subtask completion rewards and subtask values** — each node in the task hierarchy contributes to the overall value, enabling modular learning and state abstraction. **MAXQ Decomposition** - **Task Graph**: Define a directed acyclic graph of subtasks — leaf nodes are primitive actions, internal nodes are composite tasks. - **Decomposed Value**: $Q(s, a) = V(s, a) + C(s, a)$ where $V$ is the subtask's own reward and $C$ is the completion function (reward after subtask finishes). - **Recursive**: Each subtask's value decomposes further — the entire tree contributes to the root value. - **State Abstraction**: Each subtask can use its own state abstraction — only relevant features needed. **Why It Matters** - **Modularity**: Each subtask learns independently — modular, reusable value functions. - **State Abstraction**: Different subtasks can ignore irrelevant state features — faster learning. - **Interpretable**: The decomposed value function shows exactly how each subtask contributes to overall value. **MAXQ** is **value decomposition for hierarchical RL** — breaking the overall value into modular subtask contributions for efficient, interpretable learning.