Scalable Diffusion Models with Transformers

The paper explores a new class of diffusion models based on the transformer architecture, called Diffusion Transformers (DiTs). The authors train latent diffusion models of images but replace the commonly used U-Net backbone with a transformer that operates on latent patches, then analyze the models' scalability through the lens of forward-pass complexity measured in Gflops.

They find that DiTs with higher Gflops—obtained by increasing transformer depth or width or the number of input tokens—consistently achieve lower FID, indicating strong and predictable scaling behavior. Beyond good scalability, the largest model, DiT-XL/2, outperforms all prior diffusion models on the class-conditional ImageNet 512×512 and 256×256 benchmarks, achieving a state-of-the-art FID of 2.27 on the 256×256 setting, showing that transformers can serve as effective, scalable backbones for diffusion-based image generation.