Let's Verify Step by Step

Problem

Framing

Outcome-only reward models cannot localize where chain-of-thought reasoning fails, so they mis-rank solutions that reach the right answer for the wrong reasons. The paper argues that step-level process supervision fixes this credit-assignment problem and lifts best-of-$N$ math search from 72.4% to 78.2% on MATH.

Currently Used Methods

Direct antecedents

• @christianoRLHF2017 — reward modeling from human preferences for downstream policy optimization.
  • Limitation in context: feedback targets whole outputs, not individual reasoning errors.
• @ouyangInstructGPT2022 — instruction-following via RM training and RLHF.
  • Limitation in context: supervision stays outcome-level, so false-positive reasoning survives.
• @weiCoT2022 — chain-of-thought prompting improves multi-step reasoning.
  • Limitation in context: prompting exposes reasoning traces but does not verify them.
• @kojimaZeroShotCoT2022 — zero-shot step-by-step elicitation without demonstrations.
  • Limitation in context: generated steps remain unchecked and can drift logically.
• @yaoToT2023 — search over intermediate thoughts for deliberate reasoning.
  • Limitation in context: search still depends on a reliable step evaluator.

Proposed Method

Architecture

The system has two parts: a generator that emits newline-delimited solution steps, and a reward model that scores either the final answer token (ORM) or every step terminus (PRM). Large-scale models are GPT-4-derived; small-scale models use the same design with roughly $200\times$ less pretraining compute.

Loss / Objective

PRM training is next-token classification over step labels.

Here $y_i \in \{\text{pos}, \text{neg}, \text{neutral}\}$ is the label for step $i$.

Sampling Rule / Algorithm

At test time, the PRM ranks complete solutions by multiplying stepwise correctness probabilities.

Training Procedure

• PRM800K filtered training set: about $8 \times 10^5$ step labels.
• Solutions in PRM800K: about $7.5 \times 10^4$.
• Problems in PRM800K: about $1.2 \times 10^4$.
• ORM training: 100 uniform samples per problem.
• Reward models: single epoch.
• ORM: no dropout.
• Small-scale active learning pool: 1000 samples per problem.
• Active-learning mix: 80% convincing wrong-answer, 20% remaining convincing.

Evaluation

Datasets

• MATH, evaluated on a held-out 500-problem subset.
• PRM800K for step-level supervision.
• OOD STEM set: AP Calculus, AP Chemistry, AP Physics, AMC10/12.

Metrics

• % problems solved under best-of-$N$ search.
• Best-of-1860 for the main large-scale comparison.
• Best-of-100 for OOD evaluation.
• Data efficiency under varying labeled solutions per problem.

Headline results

• MATH large-scale (best-of-1860): PRM 78.2%, ORM 72.4%, majority voting 69.6%.
• MATH large-scale (all tested $N$): PRM beats ORM and majority voting.
• Small-scale synthetic supervision: process supervision beats both ORM baselines at every data scale.
• Active learning: $2.6\times$ better data efficiency than uniform labeling.
• OOD STEM aggregate (best-of-100): PRM 72.9%, ORM 63.8%, majority voting 61.3%.

Table 1: OOD generalization on recent STEM tests

Ablations

• Supervision type: process labels beat outcome labels on identical small-scale datasets.
• Outcome target source: PRM-based outcome labels beat final-answer checking.
• Data collection strategy: active learning improves sample efficiency by $2.6\times$.
• RM-weighted voting: no noticeable gain over plain PRM ranking.

Method Strengths and Weaknesses

Strengths

• Closes the main gap with a direct metric gain: 78.2% vs 72.4% best-of-1860.
• Uses dense step labels, so the reward signal localizes the first reasoning error.
• Shows robustness beyond MATH: 72.9% aggregate on fresh STEM exams.
• Releases PRM800K, a large supervised dataset for verifier research.

Weaknesses

• Main large-scale ORM and PRM training sets are not directly comparable.
• Evaluation only measures reranking, not generator improvement through RL.
• PRM score uses a simple product reduction that may over-penalize long solutions.
• Test set shrinks to 500 MATH problems because PRM800K uses many test items.

Suggestions from the authors

• Finetune the generator with RL using the reward model.
• Study process supervision in domains without easy final-answer checking.
• Improve iterative retraining for active learning without instability.
• Explore alternative reductions from step scores to solution scores.

Links

Prior Papers

• @baiConstitutionalAI2022 — another alignment paper that replaces coarse preference signals with more structured supervision.
• @yaoToT2023 — search over intermediate reasoning states becomes stronger when paired with reliable step scoring.
• @christianoRLHF2017 — establishes the reward-modeling frame that this paper refines with step-level labels.
• @ouyangInstructGPT2022 — outcome-level RLHF is the direct baseline this paper argues to surpass for reasoning.

Further Papers

• @shaoGRPO2024 — later reasoning-focused RL work that benefits from stronger reward signals for multi-step tasks.
• @rafailovDPO2023 — preference-optimization methods are adjacent alternatives to RLHF-style reward modeling.
• @kwonvLLM2023 — verifier-style reasoning supervision overlaps with this paper's process-reward modeling agenda.