dreamer
**Dreamer** is a **model-based reinforcement learning agent that achieves state-of-the-art sample efficiency by learning a world model from sensory inputs and training a policy entirely through imagined experience in the model's latent space — never requiring gradients from the real environment for policy optimization** — developed by Danijar Hafner and published in 2020 (DreamerV1), with successors DreamerV2 (2021) and DreamerV3 (2023) progressively extending to human-level Atari performance, continuous control, and a single universal hyperparameter configuration that works across radically different domains without tuning.
**What Is Dreamer?**
- **World Model**: Dreamer learns a compact latent dynamics model from visual observations — encoding pixels into vectors, predicting future latent states, and estimating rewards without ever generating pixels during imagination.
- **Imagined Rollouts**: The policy is trained entirely on imaginary trajectories generated by the world model — never touching the real environment during policy updates.
- **Actor-Critic in Imagination**: A differentiable actor and critic are trained by backpropagating through imagined sequences — gradients flow from imagined rewards back through the world model to the policy.
- **Three Learning Objectives**: (1) World model learning from real experience (reconstruct observations, predict rewards), (2) Critic learning (estimate value of imagined states), (3) Actor learning (maximize value through imagined actions).
**The RSSM Architecture**
Dreamer's world model uses the **Recurrent State Space Model (RSSM)**:
- **Deterministic path**: A GRU recurrent network maintains a deterministic recurrent state across timesteps — capturing reliable temporal context.
- **Stochastic path**: A latent variable drawn from a learned distribution captures uncertainty and environmental stochasticity at each step.
- **Prior and Posterior**: The model learns both a prior (predicting next state from action) and a posterior (inferring state from observation), trained with a KL divergence objective.
- This dual-path design captures both consistency (deterministic) and uncertainty (stochastic) — essential for modeling real environments.
**DreamerV1 → V2 → V3 Evolution**
| Version | Key Innovation | Performance |
|---------|--------------|-------------|
| **DreamerV1 (2020)** | End-to-end differentiable world model; latent imagination | 5x fewer steps than Rainbow on DMControl |
| **DreamerV2 (2021)** | Discrete latent variables; KL balancing; λ-returns | First model-based agent at human-level Atari (55/57 games) |
| **DreamerV3 (2023)** | Symlog predictions; free bits; single hyperparameter config | Works on Minecraft diamonds, robotics, tabletop, Atari without tuning |
**Why Dreamer Matters**
- **Sample Efficiency**: DreamerV3 solves Atari in 200M environment steps vs. Rainbow's 200M — but with far less wall-clock time because imagined rollouts are cheap.
- **Domain Generality**: DreamerV3's single configuration handles continuous and discrete actions, dense and sparse rewards, 2D and 3D observations — unprecedented generality.
- **Minecraft Achievement**: DreamerV3 was the first RL agent to collect diamonds in Minecraft from scratch — a long-horizon, sparse-reward benchmark considered extremely challenging.
- **Theoretical Clarity**: Dreamer provides a clean separation between world model learning and policy learning — each component is independently analyzable and improvable.
Dreamer is **the benchmark for what model-based RL can achieve** — proving that learning to imagine the future is a more powerful and efficient path to intelligent behavior than learning purely from real trial and error.