random network distillation

**RND** (Random Network Distillation) is an **exploration bonus method that detects novelty by measuring how well a predictor network can match a fixed random network's output** — novel states produce high prediction error (the predictor hasn't been trained on similar states), providing an exploration bonus. **How RND Works** - **Fixed Target**: A randomly initialized and frozen network $f_{target}(s)$ — maps states to random embeddings. - **Predictor**: A trained network $f_{predict}(s)$ — tries to match the fixed target's output. - **Novelty**: $r_i = |f_{predict}(s) - f_{target}(s)|^2$ — high error = novel state (predictor not trained on similar states). - **Training**: The predictor is trained on visited states — its error naturally decreases for familiar states. **Why It Matters** - **No Stochasticity Problem**: Unlike curiosity (ICM), RND is not confused by stochastic environments — the target is deterministic. - **Simple**: Just two networks and an MSE loss — extremely simple to implement. - **Montezuma's Revenge**: RND achieved breakthrough performance on Montezuma's Revenge — a notoriously hard-exploration Atari game. **RND** is **novelty through random targets** — detecting unfamiliar states by measuring prediction error against a fixed random network.