Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Problem

Framing

Classical search needs a simulator. Prior model-based RL learns pixels or latent reconstructions, yet still trails strong model-free agents on Atari. MuZero closes this gap by learning only reward, policy, and value signals needed for planning, then using MCTS over that latent model to reach state of the art on all 57 Atari games while matching AlphaZero-level board-game play.

Currently Used Methods

Direct antecedents

• @silverAlphaZero2018 — search plus policy-value learning with a perfect simulator.
  • Limitation in context: requires known game rules and exact transitions.
• @haWorldModels2018 — latent generative world models for control from pixels.
  • Limitation in context: plans through observation prediction, wasting capacity on irrelevant detail.
• @hafnerDreamerV1_2019 — latent dynamics optimized for imagined value learning.
  • Limitation in context: no tree search, so precise lookahead remains limited.
• "Value Prediction Networks" — action-conditioned abstract model for reward and value lookahead.
  • Limitation in context: predicts no policy targets and uses weaker search.

Proposed Method

Architecture

MuZero factorizes into representation $s^0 = h_\theta(o_{1:t})$, dynamics $(r^k, s^k) = g_\theta(s^{k-1}, a^k)$, and prediction $(p^k, v^k) = f_\theta(s^k)$. Board games and Atari use AlphaZero-style residual towers; Atari uses 16 residual blocks and all networks use 256 hidden planes.

Loss / Objective

Training matches predicted reward, value, and policy to observed reward and search-improved targets across $K$ recurrent unroll steps.

Sampling Rule / Algorithm

Action selection runs MCTS over latent states, then samples from root visit counts.

Training Procedure

• Unroll length: $K = 5$.
• Training steps: $1$M mini-batches.
• Mini-batch size: $2048$ for board games, $1024$ for Atari.
• Search simulations: $800$ per move in board games, $50$ in Atari.
• Actor checkpoint refresh: every $1000$ training steps.
• Atari reanalyse: fresh policy targets on $80\%$ of updates.

Evaluation

Datasets

• Go
• Chess
• Shogi
• Atari 2600, all 57 ALE games

Metrics

• Elo against AlphaZero in board games
• Mean human-normalized Atari score
• Median human-normalized Atari score
• Per-game mean reward over 1000 Atari episodes

Headline results

• Atari, 20B frames: mean human-normalized $4999.2\%$, median $2041.1\%$.
• Atari, 200M frames: mean human-normalized $731.1\%$, median $202.1\%$.
• Go: matches AlphaZero superhuman performance without game rules.
• Chess: matches AlphaZero superhuman performance without game rules.
• Shogi: matches AlphaZero superhuman performance without game rules.

Table 2: Header of the per-game Atari comparison table

Ablations

• Search time in Go: the learned model keeps improving with up to two orders longer search.
• Search depth in Atari: evaluation exceeds the trained $K=5$ unroll depth.
• Reanalyse targets: improves sample efficiency enough to define a second MuZero variant.
• Run-to-run Atari curves: some games are stable, others show large variance.

Method Strengths and Weaknesses

Strengths

• Removes the simulator requirement yet preserves AlphaZero-class search.
• Learns task-relevant dynamics instead of reconstructing observations.
• One algorithm spans board games and visually rich Atari.
• Search quality scales beyond the trained unroll horizon.

Weaknesses

• Still relies on expensive MCTS at acting time.
• Uses deterministic latent dynamics only.
• Atari performance shows substantial variance on some games.
• Training setup remains large-scale: $20$B Atari frames in the main setting.

Suggestions from the authors

• Extend MuZero from deterministic to stochastic dynamics.
• Apply the method to real-world domains without known rules.
• Study broader planning algorithms over the learned latent MDP.
• Improve sample efficiency of learned-model search further.

Links

Prior Papers

• @silverAlphaZero2018 — MuZero keeps AlphaZero's search-based policy iteration but replaces the simulator with learned dynamics.
• @haWorldModels2018 — both learn latent environment models, but MuZero optimizes the model for planning targets instead of observation generation.
• @hafnerDreamerV1_2019 — both use latent dynamics for control, but MuZero adds explicit tree search over the learned model.

Further Papers

• @hafnerDreamerV3_2023 — extends latent world-model RL after MuZero, emphasizing scalable representation learning and control without search.
• @hafnerDreamerV2_2021 — a close descendant in model-based RL that also learns compact latent dynamics for long-horizon decision making.