Learning Transferable Visual Models From Natural Language Supervision

This paper proposes learning visual representations directly from natural language supervision rather than fixed category labels. The method pre-trains on 400 million (image, text) pairs collected from the internet using a simple contrastive objective: predicting which caption correctly pairs with which image, a task that scales efficiently to this large, noisy web dataset.

After pre-training, natural language can be used to reference learned visual concepts or describe entirely new ones, enabling zero-shot transfer to downstream tasks without any dataset-specific training. Benchmarked across more than 30 existing computer vision datasets covering OCR, video action recognition, geo-localization, and fine-grained object classification, the model transfers non-trivially and is often competitive with fully supervised baselines, notably matching the original ResNet-50's accuracy on ImageNet in a zero-shot setting despite never seeing its 1.28 million training examples; code and pre-trained weights were released.