Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Most recent semantic segmentation systems rely on a fully-convolutional network with an encoder-decoder architecture, where the encoder progressively lowers spatial resolution while learning more abstract, semantic concepts over larger receptive fields. Because modeling context is critical for segmentation, prior efforts focused on enlarging the receptive field through dilated/atrous convolutions or attention modules, but kept the FCN encoder-decoder backbone unchanged. This paper offers an alternative perspective by casting semantic segmentation as a sequence-to-sequence prediction problem. It deploys a pure transformer, with no convolution and no resolution reduction, to encode an image as a sequence of patches, so that global context is modeled in every layer of the network.

Combining this transformer encoder with a simple decoder produces a powerful model the authors call SETR (SEgmentation TRansformer). Extensive experiments show SETR achieves new state-of-the-art results on ADE20K (50.28% mIoU) and Pascal Context (55.83% mIoU), and competitive results on Cityscapes; it also reached first place on the competitive ADE20K test-server leaderboard on the day of submission. The work helped demonstrate that transformer backbones could replace convolutional pipelines for dense prediction tasks.