Continuous control with deep reinforcement learning

Problem

Framing

Deep RL had not made off-policy value learning work in continuous action spaces with large neural function approximators. The paper closes this gap by combining deterministic policy gradients with replay, target networks, and batch normalization, then learning directly from pixels on many MuJoCo tasks and Torcs.

Currently Used Methods

Foundational

• @mnihDQN2015 — replay and target networks stabilize deep off-policy value learning.
  • Limitation in context: argmax over continuous actions is intractable.
• @schulmanTRPO2015 — constrained stochastic policy optimization for continuous control.
  • Limitation in context: on-policy updates waste data and avoid off-policy critics.
• Deterministic Policy Gradient — actor update through $\nabla_a Q$ for continuous actions.
  • Limitation in context: naive deep implementations are unstable on hard domains.
• NFQCA — neural fitted Q actor-critic with minibatch updates.
  • Limitation in context: does not stabilize large-scale pixel-based control.
• Stochastic Value Gradients — critic-based policy learning with stochastic policies.
  • Limitation in context: this paper targets simpler deterministic actors with replay.

Proposed Method

Architecture

DDPG uses an actor $\mu(\mathbf{s}\mid \theta^{\mu})$ and critic $Q(\mathbf{s},\mathbf{a}\mid \theta^Q)$, each with target copies. For low-dimensional states, both networks use two hidden layers with 400 and 300 units; the critic injects the action at the second hidden layer, and batch normalization is applied to state inputs and intermediate layers.

Loss / Objective

The critic minimizes the Bellman residual with a target-network bootstrap:

Sampling Rule / Algorithm

The actor follows the deterministic policy gradient estimated from replay-buffer states:

Target networks are updated by soft averaging:

Training Procedure

• Replay buffer: $10^6$ transitions.
• Minibatch size: $64$.
• Discount: $\gamma = 0.99$.
• Soft target rate: $\tau = 0.001$.
• Actor learning rate: $10^{-4}$.
• Critic learning rate: $10^{-3}$.
• Critic weight decay: $10^{-2}$.
• Exploration: Ornstein-Uhlenbeck noise.
• Optimizer: Adam.

Evaluation

Datasets

• Continuous-control benchmarks in MuJoCo.
• Both low-dimensional state input and pixel input settings.
• Torcs for vision-based driving.

Metrics

• Normalized reward: random policy $=0$, planner $=1$.
• Torcs raw return.
• Estimated $Q$ versus realized return diagnostics.

Headline results

• blockworld: lowd $1.156$ avg, $1.511$ best; pixels $0.466$ avg, $1.299$ best.
• cartpoleBalance: lowd $0.951$ avg, $1.000$ best; pixels $0.335$ avg, $0.996$ best.
• cheetah: lowd $1.311$ avg, $1.990$ best; pixels $1.204$ avg, $1.431$ best.
• pendulum: lowd $0.981$ avg, $1.102$ best; pixels $1.000$ avg, $1.083$ best.
• walker2d: lowd $0.705$ avg, $1.573$ best; pixels $0.944$ avg, $1.476$ best.

Table 1: Performance after training for at most 2.5 million steps; random policy is 0 and planner is 1 (except Torcs).

Ablations

• Target networks: crucial for stable learning curves.
• Batch normalization: improves actor-critic training across tasks.
• Pixel input: remains competitive with low-dimensional control on several tasks.
• Q calibration: accurate on pendulum and cartpole, worse on cheetah but still useful.

Method Strengths and Weaknesses

Strengths

• Extends replay-buffer off-policy learning to continuous actions.
• Learns directly from pixels across many continuous-control tasks.
• Soft target updates materially improve stability.
• Beats or matches planner-normalized score $1$ on several domains.

Weaknesses

• Training remains fragile; some replicas fail on hard tasks.
• Q estimates degrade on complex domains like cheetah.
• Requires many environment steps: up to $2.5$ million.
• Performance variance is large between average and best runs.

Suggestions from the authors

• Study why value estimates remain useful despite poor calibration on hard tasks.
• Improve stability and reproducibility across random seeds.
• Extend pixel-based continuous control to broader real-world domains.
• Analyze stronger exploration processes for tasks with long time scales.

Links

Prior Papers

• @mnihDQN2015 — contributes replay and target-network ideas that DDPG adapts to actor-critic control.
• @schulmanTRPO2015 — another early deep continuous-control baseline with a contrasting on-policy optimization strategy.

Further Papers

• @mnihA3C_2016 — another major deep RL control method from the same period, useful as an on-policy contrast.
• @hafnerDreamerV1_2019 — extends continuous-control learning with latent world models and actor-critic updates.
• @schulmanTRPO2015 — remains a close policy-optimization counterpart for continuous control.