options framework

**The Options Framework** is the **foundational formalism for hierarchical RL** — defining options as temporally extended actions (macro-actions) with three components: an initiation set (where the option can start), an option policy (how it acts), and a termination condition (when it finishes). **Options Formalism** - **Option $o$**: $o = (I_o, pi_o, eta_o)$ — initiation set, policy, and termination probability. - **Initiation Set $I_o$**: The set of states where option $o$ can be initiated. - **Policy $pi_o(a|s)$**: The action selection policy while option $o$ is active. - **Termination $eta_o(s)$**: Probability of terminating the option upon reaching state $s$. **Why It Matters** - **Temporal Abstraction**: Options abstract away sequences of primitive actions — enabling planning at a higher level. - **SMDP**: Options induce a Semi-Markov Decision Process (SMDP) at the higher level. - **Option-Critic**: The Option-Critic architecture learns options end-to-end using policy gradient — no manual definition needed. **The Options Framework** is **the grammar of hierarchical RL** — formalizing macro-actions as reusable, temporally extended building blocks.