Scalable Diffusion Models with Transformers
Introduces Diffusion Transformers (DiT): latent diffusion models that swap the U-Net backbone for a transformer, setting state-of-the-art ImageNet FID.
This paper introduces Diffusion Transformers (DiTs), diffusion models built on the transformer architecture. Training latent diffusion models of images, the authors replace the usual U-Net backbone with a transformer operating on latent patches, and study scalability via forward-pass complexity in Gflops. DiTs with higher Gflops—via greater depth, width, or more tokens—consistently achieve lower FID. The largest, DiT-XL/2, beats all prior diffusion models on class-conditional ImageNet 512×512 and 256×256, reaching a state-of-the-art FID of 2.27 on the latter.
Based on: Scalable Diffusion Models with Transformers · IEEE International Conference on Computer Vision
Curated by Aramai Editorial
Read summary →