Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

The paper tackles the reliance of visual and vision-language representation learning on expensively curated datasets, which caps model scale. Instead of cleaning data, the authors assemble a corpus of more than one billion image alt-text pairs obtained without the filtering and post-processing used for datasets like Conceptual Captions, and train a simple dual-encoder architecture that aligns image and text representations using a contrastive loss.

The central finding is that the scale of the noisy corpus compensates for its noise, producing state-of-the-art representations from a simple learning scheme. The visual features transfer strongly to classification tasks such as ImageNet and VTAB, while the aligned image-text representations enable zero-shot image classification, set new state-of-the-art results on Flickr30K and MSCOCO retrieval even versus more sophisticated cross-attention models, and support cross-modality search with complex text and text-plus-image queries.