Proximal Policy Optimization Algorithms

Problem

Framing

Vanilla policy gradients waste samples after one update, while TRPO stabilizes updates with a costly constrained solve. PPO closes this gap with a clipped surrogate that supports multiple minibatch epochs using first-order optimization. It reports 30/49 Atari wins by training-average reward.

Currently Used Methods

Foundational

• @mnihDQN2015 — deep Q-learning for discrete-control benchmarks.
  • Limitation in context: weak on continuous control and not direct policy optimization.
• Trust Region Policy Optimization — KL-constrained surrogate optimization for stable policy updates.
  • Limitation in context: conjugate-gradient machinery complicates implementation and scaling.
• High-Dimensional Continuous Control Using Generalized Advantage Estimation — variance-reduced advantage estimates for policy gradients.
  • Limitation in context: repeated updates on one batch still destabilize learning.
• Asynchronous Methods for Deep Reinforcement Learning — scalable actor-critic baseline for Atari.
  • Limitation in context: lower Atari sample efficiency than PPO in this comparison.
• Sample Efficient Actor-Critic with Experience Replay — strong replay-based Atari actor-critic baseline.
  • Limitation in context: similar Atari performance with more algorithmic complexity.

Proposed Method

Architecture

PPO changes the update rule, not the network family. For MuJoCo, it uses separate policy and value MLPs with two 64-unit $\tanh$ layers; the policy outputs Gaussian means with learned standard deviations. Shared policy-value parameters are also supported through a joint loss.

Loss / Objective

The core objective clips the probability ratio around the old policy.

Algorithm

PPO alternates rollout collection under $\pi_{\theta_{\mathrm{old}}}$ with several epochs of minibatch ascent on the clipped surrogate.

Training Procedure

• MuJoCo horizon: $T = 2048$
• MuJoCo Adam stepsize: $3 \times 10^{-4}$
• MuJoCo epochs: $10$
• MuJoCo minibatch size: $64$
• MuJoCo discount: $\gamma = 0.99$
• Atari horizon: $T = 128$
• Atari Adam stepsize: $2.5 \times 10^{-4} \times \alpha$
• Atari epochs: $3$
• Atari minibatch size: $32 \times 8$
• Atari discount: $\gamma = 0.99$

Evaluation

Datasets

• OpenAI Gym MuJoCo: HalfCheetah-v1, Hopper-v1, InvertedDoublePendulum-v1, InvertedPendulum-v1, Reacher-v1, Swimmer-v1, Walker2d-v1
• Atari / Arcade Learning Environment
• RoboschoolHumanoidFlagrun qualitative control

Metrics

• Continuous control: average normalized score over 21 runs
• Atari: games won by average reward over all training
• Atari: games won by average reward over last 100 episodes

Headline results

• Continuous control, no clipping: avg. normalized score $-0.39$
• Continuous control, clipping $\epsilon=0.2$: avg. normalized score $0.82$
• Continuous control, best fixed-KL baseline ($\beta=3$): avg. normalized score $0.72$
• Atari, training-average criterion: PPO wins $30/49$ games; A2C $1$, ACER $18$
• Atari, last-100-episodes criterion: PPO wins $19/49$ games; A2C $1$, ACER $28$

Table 1: Atari game wins across summary criteria

Ablations

• Clip range $\epsilon$: $0.2$ beats $0.1$ and $0.3$ on the continuous benchmark.
• Remove clipping or KL penalty: performance drops below random-policy normalization.
• KL penalty, adaptive or fixed: both trail clipping.
• Clipping in log space: no gain over ratio clipping.

Method Strengths and Weaknesses

Strengths

• Clipped updates permit multiple minibatch epochs on one rollout batch.
• First-order optimization avoids TRPO's constrained second-order solve.
• Strong continuous-control result: $0.82$ normalized score at $\epsilon=0.2$.
• Broad Atari sweep: 30/49 wins by training-average reward.

Weaknesses

• Final Atari performance trails ACER on last-100-episode reward, 19 wins versus 28.
• Performance depends on clip range; $\epsilon=0.3$ underperforms $0.2$.
• Clipping is heuristic, not a hard trust-region guarantee.
• Paper emphasizes online updates, not replay-heavy sample reuse.

Suggestions from the authors

• Analyze stronger theory for the clipped surrogate.
• Test broader architectures with parameter sharing and stochastic components.
• Extend comparisons beyond Atari and standard locomotion tasks.
• Study simpler KL-control variants that retain performance.

Links

Prior Papers

• @mnihDQN2015 — canonical deep RL baseline for the Atari comparisons.
• @christianoRLHF2017 — later preference-based RL pipelines adopt PPO-style policy optimization.
• @silverAlphaZero2018 — large-scale policy optimization in RL makes PPO a useful practical comparator.

Further Papers

• @christianoRLHF2017 — preference-based RL commonly uses PPO as the policy-update backbone.
• @ouyangInstructGPT2022 — RLHF for language models uses PPO for reward-model optimization.