rl²

**RL²** (RL-Squared, Learning to Reinforcement Learn) is a **meta-RL approach that uses a recurrent neural network to implement a learning algorithm within its activations** — the RNN's hidden state acts as a learned RL algorithm, accumulating task-specific knowledge over the course of an episode. **How RL² Works** - **Outer Loop**: Train the RNN policy across many tasks via standard RL (this is the "meta" training). - **Inner Loop**: At test time, the RNN adapts to a new task purely through its hidden state — no gradient updates. - **Input**: The RNN receives $(s_t, a_{t-1}, r_{t-1}, d_{t-1})$ — state, previous action, reward, and done flag. - **Hidden State**: The hidden state encodes the RNN's understanding of the current task — it IS the learned algorithm. **Why It Matters** - **No Gradients at Test Time**: Adaptation happens through forward passes — no backpropagation needed for new tasks. - **Learned Algorithm**: The RNN can implement sophisticated exploration strategies (e.g., Thompson sampling emerges). - **Fast**: Adaptation is as fast as a forward pass — real-time task adaptation. **RL²** is **a neural network that IS the RL algorithm** — the RNN's hidden dynamics implement a learned reinforcement learning algorithm.