DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo

2024 · arXiv

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Problem

Framing

Open open-source LLMs lag badly on competition math, and PPO-style RL adds a full value model that inflates memory cost. DeepSeekMath closes both gaps with 120B curated math tokens and GRPO, a critic-free PPO variant. The 7B RL model reaches 51.7% on MATH, rising to 60.9% with 64-sample self-consistency.

Currently Used Methods

Foundational

@brownGPT3_2020 — large-scale autoregressive pretraining establishes few-shot language modeling.
- Limitation in context: generic web data leaves math reasoning far below specialist systems.
@weiCoT2022 — chain-of-thought supervision elicits multi-step reasoning traces.
- Limitation in context: supervision alone does not close the competition-math gap.
@schulmanPPO2017 — clipped policy optimization becomes the standard RLHF optimizer.
- Limitation in context: actor-critic training doubles model memory with a value network.
@ouyangInstructGPT2022 — instruction tuning plus PPO aligns language models to preferences.
- Limitation in context: preference RL is not specialized for math correctness signals.
@rafailovDPO2023 — preference optimization removes online rollout and critic training.
- Limitation in context: offline pairwise updates weaken exploration on hard reasoning tasks.

Proposed Method

Architecture

The system has three stages: continual pretraining from DeepSeek-Coder-Base-v1.5 7B, mathematical instruction tuning, then RL with GRPO. The data pipeline iteratively trains a fastText retriever, mines Common Crawl, discovers math-heavy domains, and expands the seed set to 35.5M pages and 120B tokens.

Iterative data-mining pipeline: a fastText classifier recalls math pages from deduplicated Common Crawl, then domain discovery and URL annotation expand the seed corpus.

Loss / Objective

GRPO replaces PPO's learned baseline with group-relative normalization over sampled completions.

J_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{q \sim P(Q),\{o_i\}_{i=1}^{G} \sim \pi_{\theta_{\mathrm{old}}}(O\mid q)}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\min\left(r_{i,t}(\theta)\hat{A}_{i,t},\operatorname{clip}\left(r_{i,t}(\theta),1-\epsilon,1+\epsilon\right)\hat{A}_{i,t}\right)-\beta D_{\mathrm{KL}}\!\left(\pi_\theta\|\pi_{\mathrm{ref}}\right)\right]

r_{i,t}(\theta)=\frac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})},\qquad \hat{A}_{i}=\frac{r_i-\operatorname{mean}(\{r_j\}_{j=1}^{G})}{\operatorname{std}(\{r_j\}_{j=1}^{G})}

Sampling Rule / Algorithm

Each prompt draws a group of completions from the old policy, scores them, normalizes rewards within the group, then updates the policy with clipped ratios and a KL penalty.

\{o_i\}_{i=1}^{G} \sim \pi_{\theta_{\mathrm{old}}}(O\mid q), \qquad r_i = r_\phi(q,o_i), \qquad \hat{A}_{i}=\frac{r_i-\bar{r}}{s_r}

Training Procedure

Continual pretraining: 500B tokens.
Math corpus: 120B tokens from 35.5M web pages.
Continual pretraining mix: 56% DeepSeekMath, 4% AlgebraicStack, 10% arXiv, 20% code, 10% natural language.
Context length: 4K.
SFT: 500 steps.
SFT batch size: 256.
SFT learning rate: $5 \times 10^{-5}$ .
RL data: about 144K GSM8K and MATH questions.

Evaluation

Datasets

English math: GSM8K, MATH, SAT, OCW, MMLU-STEM.
Chinese math: MGSM-zh, CMATH, Gaokao-MathCloze, Gaokao-MathQA.
General reasoning: BBH, MMLU.
Coding: HumanEval, MBPP.
Formal math: miniF2F with Isabelle.

Metrics

Accuracy for benchmark question answering.
Top1 accuracy for single-sample decoding.
Self-consistency / majority vote at larger sample counts.
Pass@K and Maj@K for sampling analyses.

Headline results

MATH, DeepSeekMath-RL 7B: 51.7% Top1.
MATH, DeepSeekMath-RL 7B with 64-sample self-consistency: 60.9%.
GSM8K, RL gain over instruct: 82.9% → 88.2%.
MATH, RL gain over instruct: 46.8% → 51.7%.
CMATH, RL gain over instruct: 84.6% → 88.8%.

Results plot: accuracy over training steps on GSM8K and MATH, comparing RFT, Online RFT, GRPO with outcome supervision, and GRPO with process supervision.

Ablations

Math-corpus choice: DeepSeekMath Corpus dominates MathPile, OpenWebMath, and Proof-Pile-2 across GSM8K, MATH, CMATH, and BBH.
Online vs offline RL: online variants outperform offline RFT as training proceeds.
Outcome vs process supervision: GRPO with process supervision is competitive but not uniformly superior.
Iterative RL: refreshing reward models during training improves late-stage benchmark accuracy.

Method Strengths and Weaknesses

Strengths

Strong data curation: 120B math tokens let a 7B base rival far larger math models.
GRPO removes the value model, cutting PPO-style memory overhead.
RL improves both in-domain and out-of-domain math benchmarks.
Unified view of SFT, RFT, DPO, PPO, and GRPO clarifies design tradeoffs.

Weaknesses

Gains rely on a trained reward model, which can drift as policy outputs change.
RL uses only about 144K math questions, narrowing supervision diversity.
Performance still trails top closed models on MATH.
The paper mixes two contributions, making GRPO's standalone impact harder to isolate.

Suggestions from the authors

Improve reward-model generalization to out-of-distribution questions and stronger decoded solutions.
Study iterative reward-model updates as policy distributions shift during RL.
Design richer process-level rewards for long mathematical derivations.
Explore better combinations of data source, algorithm, and reward function in the unified RL paradigm.

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Problem

Framing

Currently Used Methods

Foundational

Proposed Method

Architecture

Loss / Objective

Sampling Rule / Algorithm

Training Procedure

Evaluation

Datasets

Metrics

Headline results

Ablations

Method Strengths and Weaknesses

Strengths

Weaknesses

Suggestions from the authors

Links

Prior Papers

Further Papers