Dream to Control: Learning Behaviors by Latent Imagination

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi

2019 · ICLR

Dream to Control: Learning Behaviors by Latent Imagination

Problem

Framing

World-model agents from pixels were still short-horizon: they either planned over shallow rollouts or used derivative-free policy search. Dreamer closes this gap by learning an actor-critic entirely inside latent imagination and backpropagating value gradients through the learned dynamics. On 20 visual control tasks, it reaches an average score of 823 after $5 \times 10^6$ steps.

Currently Used Methods

Foundational

@haWorldModels2018 — latent dynamics with compact rollouts for visual control.
- Limitation in context: derivative-free controller training and weaker long-horizon credit assignment.
@lillicrapDDPG2015 — deterministic actor-critic for continuous control.
- Limitation in context: learns from replay, not imagined latent trajectories.
@mnihA3C_2016 — asynchronous policy gradients for scalable online RL.
- Limitation in context: needs far more environment interaction than model-based imagination.
@haarnojaSAC2018 — entropy-regularized off-policy actor-critic.
- Limitation in context: no learned world model, so sample efficiency lags.
@schulmanPPO2017 — stable clipped policy optimization.
- Limitation in context: optimizes on real trajectories, not differentiable latent rollouts.

Proposed Method

Architecture

Dreamer combines an RSSM world model with dense actor and value heads in latent space. The world model uses an image encoder, latent transition and representation models, an image decoder, and a reward predictor; latent stochastic states are 30-dimensional diagonal Gaussians, and actor/value networks use 3 layers of 300 ELU units.

Verified architecture figure: Dreamer learns dynamics from experience, optimizes behavior in imagined latent trajectories with value gradients, then acts in the environment.

Loss / Objective

Behavior learning uses imagined $\lambda$ -returns and regresses the value head to them.

V_{\lambda}(s_{\tau}) = (1 - \lambda) \sum_{n=1}^{H-1} \lambda^{n-1} V_N^{n}(s_{\tau}) + \lambda^{H-1} V_N^{H}(s_{\tau})

V_N^{k}(s_{\tau}) = \mathbb{E}_{q_\phi, q_\theta} \left[ \sum_{n=\tau}^{h-1} \gamma^{n-\tau} r_n + \gamma^{h-\tau} v_\psi(s_h) \right], \quad h = \min(\tau + k, t + H)

\max_{\phi} \mathbb{E}_{q_\theta, q_\phi} \left[ \sum_{\tau=t}^{t+H} V_{\lambda}(s_{\tau}) \right], \qquad \min_{\psi} \mathbb{E}_{q_\theta, q_\phi} \left[ \sum_{\tau=t}^{t+H} \frac{1}{2} \left\| v_\psi(s_{\tau}) - \operatorname{sg}(V_{\lambda}(s_{\tau})) \right\|^2 \right]

Algorithm

The actor samples actions from the imagined latent state, and the value target bootstraps beyond the finite rollout horizon.

a_{\tau} \sim q_\phi(a_{\tau} \mid s_{\tau}), \qquad s_{\tau} \sim q_\theta(s_{\tau} \mid s_{\tau-1}, a_{\tau-1})

Training Procedure

Seed episodes $S = 5$ .
Action repeat $R = 2$ .
Imagination horizon $H = 15$ .
Discount $\gamma = 0.99$ .
Return parameter $\lambda = 0.95$ .
Adam learning rate $6 \times 10^{-4}$ .
Batch size $50$ sequences.
Sequence length $50$ .
Gradient clipping $100$ .
Actor entropy scale $10^{-4}$ .

Evaluation

Datasets

DeepMind Control Suite from image observations.
20 continuous-control visual tasks.
Comparisons include PlaNet, D4PG, A3C, and SLAC.

Metrics

Episode return.
Average score across tasks.
Environment steps.
Computation time.

Headline results

DeepMind Control Suite average at $5 \times 10^6$ steps: 823.
D4PG average at $10^8$ steps: 786.
Tasks solved with same hyperparameters: 20 visual control tasks.
Dreamer exceeds prior methods in data-efficiency, computation time, and final performance.

Verified results plot: imagination-horizon sweep on Cartpole Swingup, Cheetah Run, Quadruped Walk, and Walker Walk comparing Dreamer, no-value ablation, and PlaNet.

Ablations

Imagination horizon: Dreamer stays strong across $H$ , while no-value training degrades more sharply.
Value learning: adding the value head reduces shortsightedness beyond finite rollouts.
Action selection scheme: latent actor learning beats online planning on the visual control suite.
World-model objective: reconstruction-based representation learning is sufficient for strong control.

Method Strengths and Weaknesses

Strengths

Learns policy updates through analytic gradients in latent space.
Same hyperparameters work across 20 visual control tasks.
Beats D4PG average score with far fewer environment steps.
Value head makes performance less sensitive to imagination horizon.

Weaknesses

Depends on world-model accuracy for policy optimization.
Finite imagination horizon still requires bootstrap targets.
Evaluation centers on continuous control, not harder discrete domains.
Uses image reconstruction, which may spend capacity on irrelevant detail.

Suggestions from the authors

Extend latent-imagination learning to discrete-action domains like Atari and DeepMind Lab.
Improve representation objectives beyond plain reconstruction for control.
Study robustness of analytic gradient learning under model errors.
Scale world models to harder long-horizon visual tasks.

Dream to Control: Learning Behaviors by Latent Imagination

Dream to Control: Learning Behaviors by Latent Imagination

Problem

Framing

Currently Used Methods

Foundational

Proposed Method

Architecture

Loss / Objective

Algorithm

Training Procedure

Evaluation

Datasets

Metrics

Headline results

Ablations

Method Strengths and Weaknesses

Strengths

Weaknesses

Suggestions from the authors

Links

Prior Papers

Further Papers