TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

The paper proposes TransUNet, a hybrid architecture for medical image segmentation that merges the strengths of Transformers and the U-Net design. It observes that while the u-shaped U-Net architecture has become the de-facto standard for medical segmentation, the intrinsic locality of convolution operations limits its ability to explicitly model long-range dependencies, whereas Transformers offer innate global self-attention but weaker localization due to insufficient low-level detail. TransUNet uses a Transformer to encode tokenized image patches taken from a CNN feature map as an input sequence, extracting global context from the image.

The decoder then upsamples the encoded features and combines them with the high-resolution CNN feature maps, recovering localized spatial information to enable precise localization. By using Transformers as strong encoders together with U-Net's detail-preserving decoder, TransUNet achieves superior performance over various competing methods on different medical applications, including multi-organ segmentation and cardiac segmentation. The work demonstrated the value of Transformer-based global context for medical image segmentation, and its code and models were released publicly.