Adding Conditional Control to Text-to-Image Diffusion Models

The paper presents ControlNet, a neural network architecture for adding spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion model and reuses its deep and robust encoding layers, which were pretrained on billions of images, as a strong backbone for learning a diverse set of conditional controls. The trainable and locked components are joined by 'zero convolutions,' zero-initialized convolution layers whose parameters grow progressively from zero so that no harmful noise disturbs the model during finetuning.

The authors apply ControlNet to a range of conditioning signals-including edges, depth, segmentation, and human pose-on Stable Diffusion, using single or multiple conditions with or without text prompts, and find that training remains robust across both small (fewer than 50k) and large (more than 1m) datasets. By making it practical to steer powerful pretrained diffusion models with explicit spatial inputs, the method broadens the range of controllable image-generation applications.