Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Problem

Framing

Generative models had not combined normalized likelihood, exact ancestral sampling, and easy conditional manipulation in one stochastic framework. The paper closes that gap by diffusing data to a tractable terminal law, then learning only local reverse kernels. On dead leaves it reports $1.244$ bits/pixel.

Currently Used Methods

Foundational

• @hintonDeepBeliefNets2006 — greedy layerwise training for deep probabilistic models.
  • Limitation in context: no stochastic diffusion path with local reverse-step targets.
• @kingmaVAE2013 — amortized latent-variable likelihood modeling.
  • Limitation in context: depends on approximate inference, not reversible corruption dynamics.
• @goodfellowGAN2014 — adversarial generation with sharp samples.
  • Limitation in context: lacks normalized likelihood and cheap state-probability evaluation.
• @dinhNVP2017 — invertible exact-likelihood transformations.
  • Limitation in context: uses deterministic bijections, not stochastic diffusion transitions.

Proposed Method

Architecture

The model defines a forward diffusion chain $q(\mathbf{x}^{(0\cdots T)})$ and a learned reverse chain $p(\mathbf{x}^{(0\cdots T)})$. For images, the reverse model predicts per-step mean and covariance from multiscale convolutions followed by several $1 \times 1$ convolutions. Time enters through learned bump-function coefficients over $t$.

Loss / Objective

Training maximizes a variational lower bound on log likelihood.

Sampling Rule / Algorithm

Sampling starts from the tractable terminal distribution and applies learned reverse kernels for $T$ steps.

Training Procedure

• Swiss roll: $T=40$ Gaussian steps.
• Binary heartbeat: $T=2000$ binomial steps.
• Images: $T=1000$.
• Bark: $T=500$.
• Optimizer: RMSprop.
• Swiss roll MLP hidden width: $16$.
• Heartbeat MLP: $3$ hidden layers, $50$ units each.

Evaluation

Datasets

• Swiss roll
• Binary heartbeat sequences
• MNIST
• CIFAR-10
• Bark textures
• Dead leaves

Metrics

• Variational lower bound $K$
• Improvement over null model $K - L_{\mathrm{null}}$
• Bits
• Bits per sequence
• Bits per pixel

Headline results

• Swiss roll: $K=2.35$ bits, $K-L_{\mathrm{null}}=6.45$ bits.
• Binary heartbeat: $K=-2.414$ bits/seq; true process $-2.322$ bits/seq.
• Dead leaves: $1.244$ bits/pixel.
• CIFAR-10: posterior denoising and unconditional samples are shown.
• MNIST: true ancestral samples are shown.

Table 1: Log-likelihood summary across datasets.

Ablations

• Diffusion length $T$: larger $T$ makes each reverse step easier to estimate.
• Forward kernel family: Gaussian and binomial diffusions use one training framework.
• Noise schedule: likelihood depends on the hand-chosen diffusion-rate schedule.
• Task type: the same formulation covers continuous images and binary sequences.

Method Strengths and Weaknesses

Strengths

• Reduces density learning to local reverse-step estimation.
• Supports exact ancestral sampling from a normalized model.
• Handles Gaussian images and binomial sequences in one framework.
• Reports $1.184$ bits/pixel on dead leaves, close to MCGSM's $1.244$.

Weaknesses

• Sampling cost scales linearly with the full horizon $T$.
• Image backbone predates U-Net-style denoisers.
• CIFAR-10 evaluation is qualitative.
• Performance depends on hand-designed diffusion schedules.

Suggestions from the authors

• Learn diffusion-rate schedules instead of fixing them.
• Extend the framework to richer diffusion kernels.
• Use distribution multiplication for conditional inference.
• Scale reverse models for long diffusion chains.

Links

Prior Papers

• @hintonDeepBeliefNets2006 — early deep probabilistic training background for tractable generative modeling.
• @kingmaVAE2013 — variational likelihood modeling targets the same density-estimation problem.
• @goodfellowGAN2014 — contrasting route to sharp generation without normalized likelihood.
• @dinhNVP2017 — exact-likelihood generative modeling via invertible transformations.

Further Papers

• @DenoisingDiffusionProbabilisticModels2020 — turns this thermodynamic formulation into the modern denoising objective.
• @DenoisingDiffusionImplicitModels2020 — accelerates sampling within the same diffusion lineage.
• @ClassifierFreeDiffusionGuidance2022 — adds strong conditional control to diffusion generation.
• @nicholImprovedDDPM2021 — improves likelihoods and sampling efficiency for DDPM-style models.
• @songScoreSDE2020 — recasts diffusion generation as continuous-time score dynamics.