Mastering Diverse Domains through World Models

Problem

Framing

Deep RL still needs domain-specific tuning when tasks shift across action spaces, observation types, and reward scales. DreamerV3 argues that a single world-model agent can remove most retuning by making optimization scale-robust through symlog transforms, twohot value heads, KL balancing, and return-range normalization. It then outperforms specialized baselines across 150+ tasks and solves Minecraft diamond from scratch.

Currently Used Methods

Foundational

• @haWorldModels2018 — latent dynamics model for control from compact imagined rollouts.
  • Limitation in context: lacked DreamerV3's actor-critic training and cross-domain robustness.
• @hafnerDreamerV1_2019 — latent imagination with actor-critic for continuous control.
  • Limitation in context: restricted to continuous control and less robust objectives.
• @hafnerDreamerV2_2021 — discrete latents and Atari-level world-model control.
  • Limitation in context: still needed more tuning across diverse domains and scales.
• @schulmanPPO2017 — strong on-policy baseline with broad adoption.
  • Limitation in context: fixed hyperparameters trail Dreamer across diverse visual domains.
• @schrittwieserMuZero2020 — model-based value learning with strong game performance.
  • Limitation in context: more specialized and less plug-and-play across domains.

Proposed Method

Architecture

DreamerV3 keeps the Dreamer template: an RSSM world model, a critic over imagined returns, and an actor trained only in latent space. The encoder maps $\mathbf{x}_t$ to discrete $\mathbf{z}_t$; a recurrent model updates $\mathbf{h}_t$; predictions for observation, reward, and continuation are decoded from $(\mathbf{h}_t, \mathbf{z}_t)$.

Loss / Objective

The world model minimizes reconstruction, reward, continuation, dynamics, and representation losses with KL free bits and asymmetric stop-gradient placement:

Sampling Rule / Algorithm

Actor learning uses imagined trajectories and return-range normalization with a floor at $1$:

Training Procedure

• Replay capacity: $5 \times 10^6$
• Batch size: $16$
• Batch length: $64$
• Imagination horizon: $16$
• Discount: $0.997$
• Return trace parameter: $\lambda=0.95$
• Entropy scale: $\eta=3 \times 10^{-4}$
• Loss scales: $\beta_{\mathrm{pred}}=1$, $\beta_{\mathrm{dyn}}=1$, $\beta_{\mathrm{rep}}=0.1$
• Critic loss scales: $\beta_{\mathrm{val}}=1$, $\beta_{\mathrm{repval}}=0.3$
• Free bits threshold: $1$ nat
• Replay ratio: $32$
• EMA regularization rate for return scale: $0.99$

Evaluation

Datasets

• Atari: 57 tasks, 200M steps
• ProcGen: 16 tasks, 50M steps
• DMLab: 30 tasks, 100M steps
• Minecraft Diamond: 1 task, 100M steps
• Also evaluates Control Suite, Crafter, Atari 100k, and BSuite

Metrics

• Human-normalized score on Atari
• Episode return on ProcGen, DMLab, Crafter, Control Suite, BSuite
• Item-discovery rate on Minecraft Diamond
• Data-efficiency curves over environment steps

Headline results

• Atari (57-task, 200M): exceeds tuned PPO and competes with specialized expert methods.
• ProcGen (16-task, 50M): beats PPO under one fixed configuration.
• DMLab (30-task, 100M): beats prior published baselines even against some 10\times-data references.
• Minecraft Diamond (100M): first method here to collect diamonds from scratch without human data.
• Cross-domain: one hyperparameter setting works across 150+ tasks.

Ablations

• Observation symlog: removing it lowers the 14-task mean return.
• Return normalization: replacing it with advantage normalization hurts robustness.
• Symexp twohot critic/reward heads: swapping to Huber loss degrades performance.
• KL balance plus free bits: removing them destabilizes learning and lowers return.

Method Strengths and Weaknesses

Strengths

• One configuration spans discrete, continuous, visual, and vector domains.
• Latent imagination avoids expensive online planning at action time.
• Return-range normalization stabilizes exploration under sparse rewards.
• Minecraft diamond result shows unusually long-horizon capability from pixels.

Weaknesses

• Strong robustness comes from many coupled tricks, not one simple principle.
• Evaluation emphasizes aggregate breadth more than controlled per-domain analysis.
• World-model quality still bottlenecks performance on hard stochastic environments.
• Large models and long training budgets remain expensive despite single-GPU runs.

Suggestions from the authors

• Study larger world models to extend the observed scaling trends.
• Improve sparse-reward exploration beyond current entropy-based behavior learning.
• Push model-based RL to harder open-world and long-horizon domains.
• Refine robust optimization tools for broader classes of prediction targets.

Links

Prior Papers

• @haWorldModels2018 — early latent world-model control that motivates learning in imagination.
• @hafnerDreamerV1_2019 — introduces Dreamer-style latent actor-critic training.
• @hafnerDreamerV2_2021 — direct predecessor adding discrete latents and strong Atari performance.
• @schulmanPPO2017 — central fixed-hyperparameter baseline throughout the paper.
• @schrittwieserMuZero2020 — specialized model-based benchmark for comparison on hard domains.

Further Papers

No vault papers identified as further work yet.