Dreamer is a model-based reinforcement learning agent that achieves state-of-the-art sample efficiency by learning a world model from sensory inputs and training a policy entirely through imagined experience in the model's latent space — never requiring gradients from the real environment for policy optimization — developed by Danijar Hafner and published in 2020 (DreamerV1), with successors DreamerV2 (2021) and DreamerV3 (2023) progressively extending to human-level Atari performance, continuous control, and a single universal hyperparameter configuration that works across radically different domains without tuning.

What Is Dreamer?

• World Model: Dreamer learns a compact latent dynamics model from visual observations — encoding pixels into vectors, predicting future latent states, and estimating rewards without ever generating pixels during imagination.
• Imagined Rollouts: The policy is trained entirely on imaginary trajectories generated by the world model — never touching the real environment during policy updates.
• Actor-Critic in Imagination: A differentiable actor and critic are trained by backpropagating through imagined sequences — gradients flow from imagined rewards back through the world model to the policy.
• Three Learning Objectives: (1) World model learning from real experience (reconstruct observations, predict rewards), (2) Critic learning (estimate value of imagined states), (3) Actor learning (maximize value through imagined actions).

The RSSM Architecture

Dreamer's world model uses the Recurrent State Space Model (RSSM):

• Deterministic path: A GRU recurrent network maintains a deterministic recurrent state across timesteps — capturing reliable temporal context.
• Stochastic path: A latent variable drawn from a learned distribution captures uncertainty and environmental stochasticity at each step.
• Prior and Posterior: The model learns both a prior (predicting next state from action) and a posterior (inferring state from observation), trained with a KL divergence objective.
• This dual-path design captures both consistency (deterministic) and uncertainty (stochastic) — essential for modeling real environments.

DreamerV1 → V2 → V3 Evolution

Why Dreamer Matters

• Sample Efficiency: DreamerV3 solves Atari in 200M environment steps vs. Rainbow's 200M — but with far less wall-clock time because imagined rollouts are cheap.
• Domain Generality: DreamerV3's single configuration handles continuous and discrete actions, dense and sparse rewards, 2D and 3D observations — unprecedented generality.
• Minecraft Achievement: DreamerV3 was the first RL agent to collect diamonds in Minecraft from scratch — a long-horizon, sparse-reward benchmark considered extremely challenging.
• Theoretical Clarity: Dreamer provides a clean separation between world model learning and policy learning — each component is independently analyzable and improvable.

Dreamer is the benchmark for what model-based RL can achieve — proving that learning to imagine the future is a more powerful and efficient path to intelligent behavior than learning purely from real trial and error.