Consistency Models

Problem

Framing

Diffusion models deliver strong sample quality but need 10–2000 denoising steps at inference. This paper replaces iterative denoising with a self-consistent map from any PF-ODE state to the trajectory origin, enabling one-step generation while retaining multistep refinement. One-step FID reaches 3.55 on CIFAR-10 and 6.20 on ImageNet $64 \times 64$.

Currently Used Methods

Direct antecedents

• @songScoreSDE2020 — score-SDE generation via reverse-time dynamics and PF ODEs.
  • Limitation in context: still needs many solver steps for high-quality sampling.
• @karrasEDM2022 — improved diffusion design and Heun-based fast samplers.
  • Limitation in context: fast samplers still remain iterative, not one-step.
• @DenoisingDiffusionImplicitModels2020 — deterministic diffusion sampling with fewer evaluations.
  • Limitation in context: sample quality still degrades at very low NFE.
• @nicholImprovedDDPM2021 — learned-variance diffusion with fewer sampling steps.
  • Limitation in context: reduced-step diffusion is still slower than direct generation.
• @DenoisingDiffusionProbabilisticModels2020 — denoising diffusion training and ancestral sampling.
  • Limitation in context: iterative reverse chain makes inference expensive.

Proposed Method

Architecture

The model learns $f_{\theta}(\mathbf{x}, t)$, a map from any PF-ODE point to the same origin $\mathbf{x}_{\epsilon}$.It enforces the boundary condition with a skip-style parameterization

where $c_{\mathrm{skip}}(\epsilon)=1$ and $c_{\mathrm{out}}(\epsilon)=0$. Experiments reuse EDM backbones with modified skip connections.

Loss / Objective

Distillation trains neighboring PF-ODE states to produce identical outputs.

Standalone training replaces the solver pair with two Gaussian-corrupted views of the same data point.

Sampling Rule

One-step generation evaluates the model once at terminal noise. Multistep sampling alternates noise injection and consistency projection.

For distillation, adjacent PF-ODE states are linked by a one-step solver update.

Training Procedure

• PF-ODE horizon: $T=80$, $\epsilon=0.002$.
• CIFAR-10: learning rate $4\mathrm{e}{-4}$, batch size 512, $N=18$ for CD.
• ImageNet $64 \times 64$: learning rate $8\mathrm{e}{-6}$, batch size 2048, $N=40$ for CD.
• LSUN $256 \times 256$: learning rate $1\mathrm{e}{-5}$, batch size 2048, $N=40$ for CD.
• ODE solver for CD: Heun.
• EMA decay: 0.9999 on CIFAR-10; 0.999943 on ImageNet and LSUN.
• CT schedules: $s_0=2$; $s_1=150$ on CIFAR-10 and LSUN, $200$ on ImageNet; $\mu_0=0.9$ or $0.95$.

Evaluation

Datasets

• CIFAR-10
• ImageNet $64 \times 64$
• LSUN Bedroom $256 \times 256$
• LSUN Cat $256 \times 256$

Metrics

• FID
• Inception Score (CIFAR-10)
• Precision
• Recall

Headline results

• CIFAR-10, one-step CD: FID 3.55, IS 9.93.
• CIFAR-10, two-step CD: FID 2.93, IS 9.85.
• ImageNet $64 \times 64$, one-step CD: FID 6.20.
• ImageNet $64 \times 64$, two-step CD: FID 4.70.
• CT beats prior one-step non-adversarial generators on CIFAR-10, ImageNet $64 \times 64$, and LSUN.

Ablations

• Metric in CD: LPIPS beats $\ell_1$ and $\ell_2$ on CIFAR-10.
• Solver in CD: Heun beats Euler at matched $N$.
• Distillation steps $N$: larger $N$ helps until diminishing returns.
• CT schedules: adaptive $N$ and $\mu$ accelerate convergence.

Results figure

Figure 3 studies training choices on CIFAR-10: metric, solver, discretization steps, and adaptive schedules.

Table 1: Hyperparameters used for training CD and CT models

Method Strengths and Weaknesses

Strengths

• One-step generation reaches FID 3.55 on CIFAR-10.
• Two-step sampling improves further to FID 2.93.
• Retains compute-quality tradeoff through multistep sampling.
• Supports zero-shot editing without task-specific retraining.

Weaknesses

• Best distillation path still depends on a pretrained diffusion model.
• CT needs careful schedules and pretrained initialization for stability.
• Ablations show strong sensitivity to metric and solver choice.
• Theory and practice focus on one-step ODE solvers, not multistep solvers.

Suggestions from the authors

• Extend the framework from one-step to multistep ODE solvers.
• Find better time-point selection than greedy ternary search.
• Study why pretrained initialization stabilizes continuous-time CT.
• Explore stronger schedule functions for $N(\cdot)$ and $\mu(\cdot)$.

Links

Prior Papers

• @songScoreSDE2020 — supplies the score-SDE and PF-ODE foundation that consistency models collapse into direct mappings.
• @karrasEDM2022 — provides the EDM parameterization, solver choices, and teacher models used for distillation.
• @lipmanFlowMatching2022 — shares the continuous-trajectory view of generative transport with different training targets.
• @liuRectifiedFlow2022 — also learns flow-like trajectory transport toward efficient generation.
• @albergoStochasticInterpolants2023 — studies related continuous interpolation paths for generative modeling.
• @peeblesDiT2022 — a relevant contemporary diffusion architecture direction for scalable image generation.

Further Papers

No vault papers identified as further work yet.