ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

ViLBERT, short for Vision-and-Language BERT, is a model for learning task-agnostic joint representations of image content and natural language. It extends the BERT architecture into a multi-modal two-stream model that processes visual and textual inputs in separate streams, which interact through co-attentional transformer layers. The model is pretrained through two proxy tasks on the large, automatically collected Conceptual Captions dataset.

After pretraining, ViLBERT transfers to multiple established vision-and-language tasks, including visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval, by making only minor additions to the base architecture. It achieves significant improvements over existing task-specific models and reaches state of the art on all four tasks, marking a shift toward treating visual grounding as a pretrainable and transferable capability rather than something learned only during task training.