feudal networks

**Feudal Networks (FuN)** is a **hierarchical RL architecture inspired by feudalism** — a Manager network sets abstract goals in a learned latent space, and a Worker network executes primitive actions to achieve those goals, creating a two-level hierarchy of decision-making. **FuN Architecture** - **Manager**: Operates at a slower timescale — sets a goal direction $g_t$ in a learned embedding space every $c$ steps. - **Worker**: Operates at every timestep — policy is conditioned on the manager's goal: $pi_{worker}(a|s, g_t)$. - **Goal Embedding**: Goals are direction vectors in a learned state representation space — the worker should move in that direction. - **Transition Policy Gradient**: Manager is trained to set goals that lead to higher returns. **Why It Matters** - **Automatic Subgoals**: The manager learns to set meaningful subgoals — no manual subtask definition. - **Temporal Abstraction**: Manager operates at coarser timescale — handles long-horizon planning. - **State-of-Art**: FuN enabled progress on hard exploration tasks (Montezuma's Revenge) with learned hierarchies. **Feudal Networks** is **the lord-and-serf architecture** — a manager sets abstract goals, a worker executes them for flexible hierarchical RL.