Taming Transformers for High-Resolution Image Synthesis

Transformers are designed to learn long-range interactions on sequential data and continue to achieve state-of-the-art results, but unlike CNNs they contain no inductive bias prioritizing local interactions, which makes them expressive yet computationally infeasible for long sequences such as high-resolution images. The authors demonstrate how to combine the effectiveness of the CNN inductive bias with the expressivity of transformers by first using CNNs to learn a context-rich vocabulary of image constituents and then using transformers to efficiently model the composition of those constituents within high-resolution images.

The approach is readily applied to conditional synthesis, where non-spatial information such as object classes and spatial information such as segmentations can control the generated image. Notably, it produces the first results on semantically-guided synthesis of megapixel images with transformers, showing that pairing convolutional representations with transformer modeling makes high-resolution, controllable generative modeling tractable where transformers alone would be computationally infeasible.