Flamingo: a Visual Language Model for Few-Shot Learning

This paper introduces Flamingo, a family of Visual Language Models designed to adapt rapidly to novel multimodal tasks using only a handful of annotated examples. The models rely on key architectural innovations that bridge powerful pretrained vision-only and language-only models, handle sequences of arbitrarily interleaved visual and textual data, and seamlessly ingest images or videos as inputs. This flexibility lets Flamingo be trained on large-scale multimodal web corpora of interleaved text and images, which is central to endowing it with in-context few-shot learning.

Across a thorough evaluation spanning open-ended tasks like visual question answering and captioning as well as close-ended multiple-choice tasks, a single Flamingo model achieves new state-of-the-art results simply by being prompted with a few task-specific examples. On numerous benchmarks it outperforms models that were fine-tuned on thousands of times more task-specific data, demonstrating the power of few-shot prompting for multimodal models.