Diffusion Models Beat GANs on Image Synthesis

Problem

Framing

Diffusion models still trailed BigGAN-class image quality on ImageNet and LSUN. The paper closes that gap with an ablated UNet redesign plus classifier guidance, reaching ImageNet $128\times128$ FID 2.97 and guided upsampling FID 3.85 at $512\times512$.

Currently Used Methods

Foundational

• @goodfellowGAN2014 — adversarial training for high-fidelity image synthesis.
  • Limitation in context: weaker coverage and unstable training relative to diffusion.
• @DenoisingDiffusionProbabilisticModels2020 — DDPM with UNet $\epsilon$-prediction training.
  • Limitation in context: ImageNet and LSUN sample quality still trails strong GANs.
• @nicholImprovedDDPM2021 — learned variances and reduced-step diffusion sampling.
  • Limitation in context: ImageNet FID still does not beat BigGAN-deep.
• @songScoreSDE2020 — score-based conditioning links diffusion and classifier gradients.
  • Limitation in context: this paper targets stronger large-scale image synthesis quality.
• @karrasStyleGAN2019 — strong GAN baseline for photorealistic synthesis.
  • Limitation in context: lacks diffusion's likelihood training and coverage advantages.

Proposed Method

Architecture

The model keeps the DDPM UNet family and swaps in empirically stronger blocks. The final setting uses variable width, 2 residual blocks per resolution, attention at $32,16,8$, 64 channels per head, BigGAN up/down blocks, and AdaGN for timestep and class conditioning.

Loss / Objective

Training uses the improved-DDPM hybrid objective with learned reverse variances.

Sampling Rule / Algorithm

Classifier guidance shifts the reverse-step mean; for DDIM it modifies noise prediction.

Training Procedure

• Diffusion steps: $T=1000$.
• Batch size: 256 for ImageNet $128\times128$ architecture ablations.
• Batch size: 256 for ImageNet $256\times256$ guidance experiments.
• Architecture-ablation sampling steps: 250.
• Guidance-study training length: 2M iterations on ImageNet $256\times256$.
• Noise schedule: cosine for ImageNet $64\times64$; linear for $128\times128$, $256\times256$, $512\times512$.

Evaluation

Datasets

• ImageNet $64\times64$
• ImageNet $128\times128$
• ImageNet $256\times256$
• ImageNet $512\times512$
• LSUN bedroom $256\times256$
• LSUN cat $256\times256$
• LSUN horse $256\times256$

Metrics

• FID
• sFID
• Inception Score
• Precision
• Recall

Headline results

• ImageNet $128\times128$ conditional: FID 2.97.
• ImageNet $256\times256$ conditional: FID 4.59.
• ImageNet $512\times512$ conditional: FID 7.72.
• ImageNet $256\times256$ guided upsampling stack: FID 3.94.
• ImageNet $512\times512$ guided upsampling stack: FID 3.85.

Table 1: Classifier guidance on ImageNet $128\times128$ trades diversity for fidelity as gradient scale increases.

Ablations

• Width versus depth: wider models reach lower FID faster in wall-clock time.
• Attention heads: more heads or fewer channels per head improve FID.
• Attention resolutions: using $32,16,8$ beats $16$-only attention.
• BigGAN up/down blocks and AdaGN: both improve FID; residual rescaling hurts.

Method Strengths and Weaknesses

Strengths

• Beats BigGAN-class baselines on ImageNet FID across multiple resolutions.
• Guidance gives one scalar knob for fidelity versus diversity.
• Matches BigGAN-deep with as few as 25 forward passes.
• Architecture gains are cumulative across controlled ablations.

Weaknesses

• Guided sampling reduces recall as precision rises.
• Conditional synthesis needs an extra noisy-image classifier.
• Best quality still depends on many denoising steps.
• Final architecture is heavily tuned on ImageNet ablations.

Suggestions from the authors

• Develop better sample-quality metrics beyond FID and IS.
• Improve faster samplers that preserve guided-sampling quality.
• Understand why large guidance scales avoid adversarial failure.
• Leverage unlabeled pretraining before classifier-based specialization.

Links

Prior Papers

• @DenoisingDiffusionImplicitModels2020 — supplies the DDIM sampler used for low-step sampling.
• @DenoisingDiffusionProbabilisticModels2020 — establishes the DDPM framework that this paper scales and improves.
• @goodfellowGAN2014 — defines the adversarial baseline family this paper surpasses on image quality.
• @nicholImprovedDDPM2021 — contributes learned variances and fast-sampling ideas adopted here.
• @songScoreSDE2020 — provides the score-based view behind classifier-guided sampling.

Further Papers

• @ClassifierFreeDiffusionGuidance2022 — removes the external classifier while retaining guidance-based control.
• @DenoisingDiffusionImplicitModels2020 — remains a practical descendant for reduced-step diffusion sampling.
• @rombachLatentDiffusion2022 — pushes diffusion synthesis to latent spaces for cheaper high-resolution generation.