Zero-Shot Text-to-Image Generation

The paper addresses text-to-image generation, which had traditionally focused on finding better modeling assumptions for training on a fixed dataset, often involving complex architectures, auxiliary losses, or side information such as object part labels or segmentation masks supplied during training. Instead, the authors propose a simple approach based on a transformer that autoregressively models the text and image tokens together as a single stream of data, removing the need for such task-specific machinery.

The key finding is that with sufficient data and model scale, this simple autoregressive transformer becomes competitive with previous domain-specific text-to-image models even when evaluated in a zero-shot fashion, without training on the target dataset. This showed that a unified token-stream formulation paired with scale could match specialized architectures that rely on complex components, auxiliary losses, or side information.