Masked-attention Mask Transformer for Universal Image Segmentation

Image segmentation groups pixels by different semantics, such as category or instance membership, and each choice of semantics defines a distinct task like panoptic, instance, or semantic segmentation. Rather than designing a specialized architecture for each, this paper presents the Masked-attention Mask Transformer (Mask2Former), a single architecture capable of addressing any of these tasks. Its central innovation is masked attention, which extracts localized features by constraining cross-attention to lie within predicted mask regions.

Mask2Former reduces research effort by at least a factor of three by unifying tasks that previously required separate models, while also outperforming the best specialized architectures by a significant margin on four popular datasets. It set new state-of-the-art results for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO), and semantic segmentation (57.7 mIoU on ADE20K), showing a universal design can beat task-specific ones.