Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

The Pyramid Vision Transformer (PVT) is a simple, convolution-free backbone network aimed at many dense prediction tasks. Unlike the Vision Transformer (ViT), designed specifically for image classification and yielding low-resolution outputs at high computational and memory cost, PVT can be trained on dense partitions of an image to achieve high output resolution and uses a progressive shrinking pyramid to reduce the computation of large feature maps, inheriting the advantages of both CNNs and Transformers.

This makes PVT a unified backbone that can serve as a direct replacement for CNN backbones across various vision tasks without convolutions. Extensive experiments validate it on object detection and instance and semantic segmentation: with a comparable number of parameters, PVT+RetinaNet achieves 40.4 AP on the COCO dataset, surpassing ResNet50+RetinaNet's 36.3 AP by 4.1 absolute AP, positioning PVT as a useful alternative backbone for pixel-level prediction.