Mastering the Game of Go with Deep Neural Networks and Tree Search

Problem

Framing

Go resists brute-force search because position evaluation is weak and legal move branching is huge. The paper closes this by coupling deep policy and value networks with Monte Carlo tree search, then reports a 5–0 match win over Fan Hui and wins against leading Go programs.

Currently Used Methods

Foundational

• @mnihDQN2015 — deep RL couples value learning with search-free action selection.
  • Limitation in context: no tree search, and no method for Go's extreme branching.

Proposed Method

Architecture

AlphaGo uses three networks around MCTS: a supervised policy $p_{\sigma}(a \mid s)$, an RL policy $p_{\rho}(a \mid s)$, and a value network $v_{\theta}(s)$. The policy and value nets are 13-layer convolutional models over board-state feature planes; the policy outputs move probabilities over legal moves, while the value net outputs a scalar win estimate.

Loss / Objective

The pipeline trains three objectives in sequence.

Algorithm

Search selects actions by action value plus an exploration bonus from the policy prior, then backs up a mixed value from the value net and rollouts.

Training Procedure

• Supervised policy data: 30 million expert positions.
• RL policy: self-play policy-gradient updates.
• Value network data: 30 million self-play positions.
• Network depth: 13 convolutional layers.
• Distributed search hardware: 1,202 CPUs and 176 GPUs.
• Single-machine search hardware: 40 search threads, 48 CPUs, 8 GPUs.

Evaluation

Datasets

• Human expert games for supervised policy training.
• Self-play games for RL policy improvement.
• Self-play positions with outcomes for value learning.
• Tournament games against Pachi, Fuego, Crazy Stone, GnuGo, and Fan Hui.

Metrics

• Move prediction accuracy.
• Win rate against baseline programs.
• Elo rating.
• KGS rank.
• Match score against Fan Hui.

Headline results

• Human-expert move prediction: 57.0% accuracy for the supervised policy network.
• Policy-only play: RL policy beats the supervised policy in head-to-head evaluation.
• Value-only evaluation: value net outperforms value derived from the supervised policy.
• Tournament play: AlphaGo defeats Fan Hui 5–0 in the formal match.
• Program strength: distributed AlphaGo surpasses prior Go engines by large win-rate margins.

Ablations

• Policy strength: more filters raise both move accuracy and playing strength.
• Search components: value net and rollouts are complementary inside MCTS.
• Training source: RL policy plays stronger than the supervised policy.
• Hardware scale: distributed search is much stronger than the single-machine version.

Method Strengths and Weaknesses

Strengths

• Marries learning and search in one system, instead of relying on handcrafted Go evaluation.
• Uses self-play to improve beyond imitation of human expert moves.
• Value network cuts rollout dependence by evaluating positions directly.
• Demonstrates real match strength with a formal 5–0 result against Fan Hui.

Weaknesses

• Search still needs heavy compute: distributed play uses 1,202 CPUs and 176 GPUs.
• Training is staged, with separate SL, RL, and value phases.
• Performance depends on MCTS; the raw networks alone are weaker.
• The system remains domain-specific to Go state and move structure.

Suggestions from the authors

• Reduce search cost while keeping the strength gain from policy and value guidance.
• Extend the learning-plus-search recipe to other perfect-information games.
• Improve neural evaluation so fewer rollout simulations are needed.
• Replace handcrafted search components with stronger learned components.

Links

Prior Papers

• @mnihDQN2015 — establishes deep reinforcement learning as a practical control paradigm that AlphaGo extends with search.

Further Papers

• @christianoRLHF2017 — extends policy improvement with human-guided preference signals instead of game outcomes alone.
• @silverAlphaZero2018 — removes human expert pretraining and learns superhuman play from self-play alone.