Trust Region Policy Optimization

Problem

Framing

Policy-gradient updates were brittle: large steps collapsed performance, small steps wasted samples. TRPO closes this gap by maximizing a local surrogate under an explicit KL trust-region constraint, giving monotonic-improvement style guarantees and strong results on MuJoCo locomotion and Atari.

Currently Used Methods

Foundational

• @mnihDQN2015 — deep Q-learning for Atari from pixels.
  • Limitation in context: value-based and discrete-action; not a stable stochastic-policy update rule.
• @mnihA3C_2016 — asynchronous actor-critic with parallel on-policy learning.
  • Limitation in context: no explicit trust region, so update size remains hard to control.
• @lillicrapDDPG2015 — deterministic actor-critic for continuous control.
  • Limitation in context: off-policy instability and no monotonic improvement constraint.
• @schulmanPPO2017 — clipped surrogate that approximates trust-region updates.
  • Limitation in context: relaxes the hard KL constraint that TRPO enforces.
• @haarnojaSAC2018 — entropy-regularized off-policy actor-critic.
  • Limitation in context: trades guarantees for efficiency; not the trust-region formulation.

Proposed Method

Architecture

TRPO is an update rule, not a new policy network. The paper instantiates it with small MLP policies for locomotion and a convolutional network for Atari.

Loss / Objective

The update maximizes a first-order surrogate while constraining average policy drift.

Algorithm

TRPO solves the constrained step with a quadratic KL model, conjugate gradient, then backtracking line search.

Training Procedure

• MuJoCo policy parameters: 364 / 4806 / 8206 for Swimmer / Hopper / Walker.
• MuJoCo simulator steps per iteration: 5{,}000 / 25{,}000 / 50{,}000.
• Conjugate gradient iterations: $10$.
• Atari policy parameters: $33{,}500$.
• Atari vine simulator steps per iteration: $400\mathrm{K}$.
• Atari single-path simulator steps per iteration: $100\mathrm{K}$.

Evaluation

Datasets

• MuJoCo locomotion: Swimmer, Hopper, Walker.
• Atari from pixels: Beam Rider, Breakout, Enduro, Pong, Q*bert, Seaquest, Space Invaders.

Metrics

• Episodic return for locomotion.
• Game score for Atari.
• Learning curves over policy iterations.
• Average KL-constrained policy improvement.

Headline results

• Swimmer: vine and single-path reach about $0.14$ reward, above natural gradient and CEM.
• Hopper: vine reaches about $2.2$ reward, above single-path near $2.1$.
• Walker: vine reaches about $2.9$ reward, above single-path near $2.7$.
• Cartpole: trust-region variants reach near-optimal reward in about $20$--$40$ iterations.
• Atari: TRPO variants are competitive with prior deep RL baselines across seven games.

Ablations

• Sampling procedure: vine reduces variance and improves learning on harder control tasks.
• Trust-region choice: fixed KL works better than penalty-based natural gradient tuning.
• Domain: gains persist from low-dimensional states to pixel-based Atari control.
• Baseline class: TRPO beats derivative-free methods on sample efficiency.

Method Strengths and Weaknesses

Strengths

• Hard KL constraint directly controls destructive policy jumps.
• Monotonic-improvement analysis motivates the surrogate objective.
• Works for both continuous control and pixel-based Atari.
• Conjugate-gradient solver avoids explicit Fisher matrix inversion.

Weaknesses

• Second-order step requires repeated Fisher-vector products.
• Line search and conjugate gradient add implementation complexity.
• Vine sampling needs state reset access; many environments cannot provide it.
• Atari error bars are absent because each task is run once.

Suggestions from the authors

• Extend the monotonic-improvement analysis to broader policy classes.
• Reduce computation in Fisher-vector products and conjugate-gradient solves.
• Study lower-variance sampling without simulator state resets.
• Improve scalability on high-dimensional vision-based control.

Links

Prior Papers

• @mnihDQN2015 — establishes the Atari deep RL benchmark that TRPO later tests against.
• @lillicrapDDPG2015 — provides a contemporary continuous-control actor baseline for comparison.
• @mnihA3C_2016 — represents early scalable policy-gradient actor-critic methods without trust-region constraints.

Further Papers

• @schulmanPPO2017 — simplifies TRPO's constrained update into a clipped first-order objective.
• @haarnojaSAC2018 — advances policy optimization in continuous control with off-policy entropy regularization.
• @hafnerDreamerV1_2019 — applies policy optimization inside a learned world model for sample-efficient control.
• @hafnerDreamerV2_2021 — extends model-based actor learning to Atari-scale pixel control.
• @hafnerDreamerV3_2023 — modern descendant on large-scale general-control benchmarks with learned dynamics.