Adding Conditional Control to Text-to-Image Diffusion Models

Problem

Framing

Spatial conditions like edges, pose, or depth were hard to inject into large text-to-image diffusion models without destroying pretrained generation quality. ControlNet closes this gap by cloning locked U-Net blocks into trainable branches linked by zero-initialized convolutions, yielding robust control across canny, depth, segmentation, and pose conditions.

Currently Used Methods

Foundational

• @rombachLatentDiffusion2022 — latent diffusion enables strong text-to-image synthesis with frozen pretrained backbones.
  • Limitation in context: lacks a dedicated pathway for spatial condition injection.
• @DenoisingDiffusionProbabilisticModels2020 — denoising objective for iterative generative modeling.
  • Limitation in context: does not address structured control signals like pose or edges.
• @nicholImprovedDDPM2021 — improved diffusion training and sampling for higher-quality generation.
  • Limitation in context: improves generation, not condition fidelity under spatial guidance.
• @ClassifierFreeDiffusionGuidance2022 — mixes conditional and unconditional scores for prompt guidance.
  • Limitation in context: text guidance alone cannot enforce pixel-aligned structure.
• @ronnebergerUNet2015 — encoder–decoder with skip connections for dense image prediction.
  • Limitation in context: no locked-copy design for safe finetuning of diffusion backbones.

Proposed Method

Architecture

ControlNet augments each pretrained Stable Diffusion encoder block and middle block with a locked copy plus a trainable copy. The trainable branch consumes condition features and merges back through zero convolutions, so control starts at exactly zero and grows during finetuning.

Loss / Objective

ControlNet keeps the base diffusion objective and optimizes the conditional denoiser with text condition $c_t$ and spatial condition $c_f$:

Sampling Rule

Sampling reuses the Stable Diffusion reverse process with the ControlNet-augmented noise predictor:

Training Procedure

• Base model: Stable Diffusion finetuning with locked backbone copy
• Control connectors: zero-initialized $1 \times 1$ convolutions
• Prompt dropout: 50% text prompts replaced by empty strings
• Cost: about 23% more GPU memory than Stable Diffusion finetuning
• Regime: trained on both small-scale and large-scale condition datasets

Evaluation

Datasets

• COCO
• ADE20K
• OpenPose-derived human pose conditions
• Depth, normal, canny, HED, M-LSD, scribble, and sketch condition sets
• Small custom condition datasets and large-scale general-domain data

Metrics

• Average User Ranking (AUR) for result quality
• Average User Ranking (AUR) for condition fidelity
• Reconstructed IoU for segmentation controllability
• FID
• CLIP-score
• CLIP-aesthetic score

Headline results

• User study, sketch control: ControlNet ranks first in both image quality and condition fidelity.
• ADE20K segmentation: reconstructed IoU exceeds LDM and PITI baselines.
• Image quality benchmark: ControlNet improves CLIP-score and CLIP-aesthetic over Stable Diffusion.
• Robustness: usable control appears with small datasets and scales further with millions of images.

Table 1: Image quality metrics comparing Stable Diffusion variants

Ablations

• Zero convolutions: stabilize finetuning by preventing harmful initialization shocks.
• Dataset size: 1k images already teaches control; 50k and 3M sharpen reliability.
• Prompt dropout: empty-text training improves no-prompt controllability.
• Architecture choice: locked backbone plus trainable copy beats naive direct finetuning.

Method Strengths and Weaknesses

Strengths

• Adds spatial control without retraining the whole text-to-image model.
• Zero-initialized connectors make finetuning stable from pretrained weights.
• Works across many condition types with one architectural recipe.
• Improves FID, CLIP-score, and CLIP-aesthetic over Stable Diffusion.

Weaknesses

• Adds about 23% training-time GPU memory over plain Stable Diffusion finetuning.
• Quantitative comparisons are concentrated on a few control settings.
• Quality depends on external condition extractors such as pose or edge detectors.
• Extra control branches increase engineering complexity at deployment.

Suggestions from the authors

• Transfer pretrained ControlNets to other diffusion backbones and stylistic checkpoints.
• Study richer combinations of multiple conditions in one generation pass.
• Scale training data further to probe controllability gains.
• Explore more task-specific adapters for additional spatial modalities.

Links

Prior Papers

• @rombachLatentDiffusion2022 — ControlNet is a condition-injection finetuning scheme built directly on Stable Diffusion's latent U-Net.
• @DenoisingDiffusionProbabilisticModels2020 — the training objective remains the standard diffusion noise-prediction loss.
• @nicholImprovedDDPM2021 — ControlNet inherits the improved DDPM-style denoising framework used in modern latent diffusion.
• @ClassifierFreeDiffusionGuidance2022 — prompt dropout with empty text connects directly to classifier-free guidance practice.

Further Papers

• @songConsistencyModels2023 — another 2023 diffusion paper that targets practical generation improvements with altered inference behavior.
• @peeblesDiT2022 — extends the diffusion model design space beyond U-Nets, relevant for future controlled backbones.
• @karrasEDM2022 — refines diffusion training and sampling, complementary to ControlNet-style conditioning modules.
• @luDPMSolver2022 — accelerates diffusion sampling and naturally pairs with controlled generation pipelines.