Video Diffusion Models

Problem

Framing

Diffusion models had not shown strong, temporally coherent video generation across unconditional, predictive, and text-conditioned settings. The paper closes this gap with a 3D U-Net video diffusion model plus reconstruction-guided conditional sampling, reaching UCF101 FID $295 \pm 3$ and BAIR FVD $66.92$.

Currently Used Methods

Foundational

• @oordWaveNet2016 — autoregressive likelihood modeling with strong local structure.
  • Limitation in context: sequential decoding makes long coherent video generation slow.
• @DenoisingDiffusionProbabilisticModels2020 — denoising diffusion for high-quality image generation.
  • Limitation in context: operates on images, not factorized space-time video tensors.
• @songScoreSDE2020 — score-based generative modeling with predictor-corrector sampling.
  • Limitation in context: no video architecture or video-specific conditional extension rule.
• @nicholImprovedDDPM2021 — improved diffusion training and faster sampling variants.
  • Limitation in context: does not address temporal coherence or video conditioning.
• @ClassifierFreeDiffusionGuidance2022 — guidance without an external classifier for conditional diffusion.
  • Limitation in context: alone it does not solve long-horizon video extension.

Proposed Method

Architecture

The model is a 3D U-Net over tensors shaped as frames $\times$ height $\times$ width $\times$ channels. It replaces each 2D convolution with a space-only 3D convolution, then factorizes attention into spatial attention and temporal attention with relative position embeddings. Conditioning $c$ and log-SNR $\lambda_t$ are injected through residual-block embeddings.

Loss / Objective

Training uses the standard denoising regression objective over noisy videos:

Sampling Rule / Algorithm

The ancestral sampler starts from $\mathbf{z}_1 \sim \mathcal{N}(0, \mathbf{I})$ and steps from $t$ to $s < t$ by

Training Procedure

• Optimizer: Adam, $\beta_1 = 0.9$, $\beta_2 = 0.99$.
• Learning rate: $2 \times 10^{-4}$ or $3 \times 10^{-4}$.
• Batch size: 128 or 256.
• EMA: 0.9999.
• Dropout: 0.0 or 0.1.
• Attention resolutions: 8, 16, 32.
• Attention head dimension: 64 or 128.
• Conditioning embedding dimension: 1024.

Evaluation

Datasets

• UCF101: unconditional generation, 16 frames at $64 \times 64$.
• BAIR Robot Pushing: predict 15 frames from 1 frame at $64 \times 64$.
• Kinetics-600: predict 11 frames from 5 frames at $64 \times 64$.
• Text-to-video dataset: 10 million captioned videos.

Metrics

• FVD.
• FID.
• IS.
• FID-avg / IS-avg: frame-averaged image metrics.
• FID-first / IS-first: first-frame image metrics.

Headline results

• UCF101 unconditional: FID $295 \pm 3$, IS $57 \pm 0.62$.
• BAIR prediction: FVD $66.92$ with Langevin sampler, 256 steps.
• Kinetics-600 prediction: FVD $16.2 \pm 0.34$, IS $15.64$ with Langevin sampler, 128 steps.
• Text-to-video, joint training with 8 image frames: FVD $57.84/60.72$, FID-avg $15.57/15.44$.
• Text-to-video, guidance weight 2.0, frameskip 1: FID-avg $10.53/10.47$, IS-avg $13.22/12.10$.

Table 1: Unconditional video modeling results on UCF101.

Ablations

• Joint image-video training: adding 4 or 8 image frames sharply improves FVD and FID.
• Classifier-free guidance: IS rises with guidance weight, while FID improves then worsens.
• Reconstruction guidance vs replacement: reconstruction guidance preserves temporal coherence and lowers FVD.
• Sampler choice: Langevin sampling beats ancestral sampling on BAIR and Kinetics-600.

Method Strengths and Weaknesses

Strengths

• Reaches UCF101 FID $295 \pm 3$, far ahead of prior reported results.
• Strong conditional prediction results on BAIR and Kinetics-600.
• Joint image-video training materially improves text-to-video sample quality.
• Reconstruction guidance fixes temporal incoherence from replacement conditioning.

Weaknesses

• Long videos require block-autoregressive extension, not single-pass generation.
• Text-conditioned metrics still degrade at high guidance weights.
• Reported training uses large TPU-v4 budgets.
• Core text-to-video architecture details sit in Appendix A, not main method.

Suggestions from the authors

• Explore training with independent images from larger image-only datasets.
• Improve conditional extension for longer and higher-resolution videos.
• Scale text-conditioned video generation beyond the initial reported results.
• Study better evaluation protocols and preprocessing consistency for video metrics.

Links

Prior Papers

• @oordWaveNet2016 — early autoregressive generative modeling baseline for sequential data, relevant as a pre-diffusion alternative.
• @DenoisingDiffusionProbabilisticModels2020 — establishes the core diffusion framework extended here from images to video.
• @songScoreSDE2020 — provides score-based sampling ideas that connect to the predictor-corrector sampler used here.
• @nicholImprovedDDPM2021 — improves diffusion training and sampling, forming part of the immediate image-diffusion foundation.
• @ClassifierFreeDiffusionGuidance2022 — guidance mechanism directly evaluated here for text-conditioned video generation.

Further Papers

• @rombachLatentDiffusion2022 — extends diffusion generation to more efficient latent spaces, relevant for scaling video models.
• @peeblesDiT2022 — explores transformer backbones for diffusion, a natural architectural successor to 3D U-Nets.
• @songConsistencyModels2023 — targets faster diffusion-style sampling, relevant to the slow multi-step generation bottleneck.
• @karrasEDM2022 — refines diffusion design and sampling choices that could improve video diffusion quality and efficiency.