ham

**HAM** (Hierarchies of Abstract Machines) is a **hierarchical RL framework that constrains the agent's policy space using partial programs** — defining the high-level task structure as a set of abstract machines (finite state controllers) that specify the skeleton of behavior, with choice points where RL selects among alternatives. **HAM Components** - **Abstract Machines**: Finite state machines that define the structure of behavior for each subtask. - **Choice Points**: States in the abstract machine where RL must decide which sub-machine to call or which action to take. - **Call Stack**: HAMs can call other HAMs — creating a hierarchical call structure (like function calls). - **Constrained MDP**: The HAM reduces the original MDP to a constrained SMDP over just the choice points. **Why It Matters** - **Domain Knowledge**: HAMs encode domain knowledge as program structure — RL only fills in the decisions. - **Reduced Search**: By constraining the policy space, HAMs dramatically reduce the RL search problem. - **Composable**: HAMs compose hierarchically — complex behaviors emerge from combining simple machines. **HAM** is **programming the structure, learning the decisions** — using abstract machines to constrain hierarchical RL with domain knowledge.