DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps

Problem

Framing

Diffusion models delivered strong likelihoods and sample quality, but training-free samplers still needed about $50+$ network evaluations for high fidelity. DPM-Solver closes this gap by solving the diffusion ODE with an exponential-integrator view that treats the linear term exactly, reaching CIFAR-10 FID $4.70$ in $10$ NFE and $2.87$ in $20$ NFE.

Currently Used Methods

Direct antecedents

• @DenoisingDiffusionProbabilisticModels2020 — discrete-time diffusion with ancestral sampling.
  • Limitation in context: plain sampling needs hundreds to thousands of evaluations.
• @DenoisingDiffusionImplicitModels2020 — deterministic non-Markovian sampler for fewer steps.
  • Limitation in context: sample quality still degrades sharply in the $10$-step regime.
• @songScoreSDE2020 — continuous-time SDE/ODE view with black-box solvers.
  • Limitation in context: ignores diffusion ODE semi-linearity, wasting solver accuracy.
• @nicholImprovedDDPM2021 — improved schedules and learned variance for faster diffusion sampling.
  • Limitation in context: training-free high-fidelity sampling still needs many evaluations.
• @dhariwalDiffusionBeatGANs2021 — classifier-guided diffusion with strong image quality.
  • Limitation in context: guided sampling remains too slow for practical deployment.

Proposed Method

Architecture

The method adds no new network. It wraps a pretrained noise predictor $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)$, discrete- or continuous-time, inside a dedicated ODE solver in half-logSNR $\lambda$. The key structural claim is that diffusion ODEs are semi-linear: the data term is integrated analytically, and only an exponentially weighted neural-network integral is approximated.

Loss / Objective

Sampling is training-free; the pretrained model still uses the standard noise-prediction loss.

Sampling Rule / Algorithm

The solver starts from the exact diffusion-ODE solution, then approximates only the weighted network integral.

Its first-order update is:

Training Procedure

• Extra training: none.
• Solver orders: $1$, $2$, $3$.
• Uniform schedule: split $[\lambda_T,\lambda_0]$ uniformly.
• Default sampling: uniform schedule for NFE $<20$.
• Default sampling: DPM-Solver-3 with adaptive schedule for NFE $\geq 20$.
• Adaptive tolerances: $\mathrm{rtol}=0.05$, $\mathrm{atol}=0.0078$, $h_{\mathrm{init}}=0.05$, $\theta=0.9$.
• DPM-Solver-2 uses $r_1=0.5$.
• DPM-Solver-3 uses $r_1=\frac{1}{3}$, $r_2=\frac{2}{3}$.

Evaluation

Datasets

• CIFAR-10.
• CelebA $64\times64$.
• ImageNet $64\times64$.
• ImageNet $128\times128$ with classifier guidance.
• LSUN bedroom $256\times256$.
• ImageNet $256\times256$ qualitative samples.

Metrics

• FID on $50\mathrm{K}$ generated samples.
• NFE as sampling cost.
• Wall-clock runtime per batch in the appendix.

Headline results

• CIFAR-10 continuous: FID $4.70$ at $10$ NFE, $2.87$ at $20$ NFE.
• CIFAR-10 discrete: DPM-Solver is best across the plotted few-step regime; about $3$ FID by $15$-$20$ NFE.
• CelebA $64\times64$: about $3$ FID at $15$-$20$ NFE, ahead of DDIM and DDPM.
• ImageNet $128\times128$ guided: strong few-step gains over DDIM at $10$-$20$ NFE.
• Multiple datasets: reported $4\times$-$16\times$ speedup over prior training-free samplers.

Table 1: FID on CIFAR-10 for different orders of RK methods and DPM-Solvers, varying the number of function evaluations (NFE).

Ablations

• Solver order: higher order helps most under tight NFE budgets.
• Step variable: uniform spacing in $\lambda$ beats uniform spacing in $t$.
• Adaptive schedule: preferred once NFE reaches about $20+$.
• Conditional sampling: the same solver remains effective with classifier guidance.

Method Strengths and Weaknesses

Strengths

• Exact treatment of the linear ODE part removes a major discretization source.
• Training-free drop-in sampler for both discrete-time and continuous-time DPMs.
• Third-order solver gives strong quality in $10$-$20$ NFE.
• Works with classifier guidance and large-scale image models.

Weaknesses

• Quality still depends on the pretrained score model.
• Few-step gains come from deterministic ODE sampling, not stochastic correction.
• Higher-order variants need extra intermediate evaluations and bookkeeping.
• Theory covers convergence order, not full optimality under guidance mismatch.

Suggestions from the authors

• Study solvers of order $k\geq 4$ with more intermediate points.
• Improve conditional sampling, especially under stronger guidance.
• Refine adaptive step-size control for better NFE-accuracy trade-offs.
• Extend dedicated solvers to broader diffusion formulations and schedules.

Links

Prior Papers

• @DenoisingDiffusionProbabilisticModels2020 — establishes the discrete-time diffusion formulation that DPM-Solver accelerates at sampling time.
• @DenoisingDiffusionImplicitModels2020 — provides the main deterministic few-step baseline that DPM-Solver substantially outperforms.
• @songScoreSDE2020 — gives the SDE/ODE unification that DPM-Solver specializes with a dedicated semi-linear solver.
• @nicholImprovedDDPM2021 — improves practical diffusion parameterizations and schedules that DPM-Solver can sample faster.
• @dhariwalDiffusionBeatGANs2021 — supplies the guided ImageNet diffusion models used to showcase few-step conditional sampling.

Further Papers

• @karrasEDM2022 — studies improved diffusion design choices and samplers in the same efficiency-quality space.
• @rombachLatentDiffusion2022 — makes diffusion sampling efficiency more consequential by moving generation into latent space.
• @peeblesDiT2022 — large diffusion transformers benefit directly from stronger few-step solvers.
• @songConsistencyModels2023 — pushes the same agenda further by collapsing diffusion sampling to very few steps.