Self-Attention Generative Adversarial Networks

The Self-Attention Generative Adversarial Network (SAGAN) brings attention-driven, long-range dependency modeling to image generation tasks. Traditional convolutional GANs generate high-resolution details as a function only of spatially local points in lower-resolution feature maps, whereas SAGAN can generate details using cues from all feature locations. Its discriminator can likewise verify that highly detailed features in distant portions of the image are consistent with one another. Building on the insight that generator conditioning affects performance, the authors apply spectral normalization to the generator to improve its training dynamics.

SAGAN achieves state-of-the-art results on the challenging ImageNet dataset, boosting the best published Inception score from 36.8 to 52.52 and reducing the Frechet Inception distance from 27.62 to 18.65. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape, indicating that self-attention helps the model capture global structure in generated images.