BEiT: BERT Pre-Training of Image Transformers

BEiT (Bidirectional Encoder representation from Image Transformers) brings BERT-style self-supervised pretraining to vision Transformers via a masked image modeling objective. Each image is represented in two views: raw image patches such as 16x16 pixels, and discrete visual tokens obtained by first tokenizing the original image. During pretraining, some image patches are randomly masked and passed through the backbone Transformer, which must recover the original visual tokens from the corrupted input. The pretrained encoder is then fine-tuned on downstream tasks by appending task-specific layers.

On image classification and semantic segmentation, BEiT achieves results competitive with prior pretraining methods. Base-size BEiT reaches 83.2% top-1 accuracy on ImageNet-1K, significantly surpassing from-scratch DeiT training at 81.8% under the same setup, while large-size BEiT attains 86.3% using only ImageNet-1K, even outperforming ViT-L supervised-pretrained on ImageNet-22K at 85.2%. This demonstrated that masked-token self-supervision could rival or exceed supervised pretraining for vision Transformers, with code and models released publicly.