curiosity-driven learning

**Curiosity-Driven Learning** is a **specific form of intrinsic motivation where the agent is rewarded for encountering situations that are difficult to predict** — the agent's curiosity reward is the prediction error of a forward dynamics model, driving it toward novel, surprising states. **ICM (Intrinsic Curiosity Module)** - **Forward Model**: Predicts next state features: $hat{phi}(s_{t+1}) = f(phi(s_t), a_t)$. - **Curiosity Reward**: $r_i = |hat{phi}(s_{t+1}) - phi(s_{t+1})|^2$ — prediction error = surprise. - **Feature Space**: Predict in a learned feature space, not raw pixels — avoids the "noisy TV" problem. - **Inverse Model**: Predict action from consecutive states — ensures the feature space captures actionable information. **Why It Matters** - **No Reward Needed**: The agent explores effectively driven purely by curiosity — no external reward required. - **Game Playing**: Curiosity-driven agents learn to play Atari games with zero external reward — remarkable emergent behavior. - **Transfer**: Curiosity-learned representations transfer to downstream tasks. **Curiosity-Driven Learning** is **exploring the unpredictable** — rewarding the agent for encountering states it cannot yet predict.