Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

The paper introduces Visual Genome, a dataset built to move computer vision beyond perceptual recognition toward cognitive tasks such as image description and question answering. The authors argue that answering a question like 'What vehicle is the person riding?' requires identifying objects plus the relationships between them, for example that a man is riding a carriage and a horse is pulling it. Because models for such tasks were still being trained on datasets designed for perceptual tasks, they collect dense annotations of objects, attributes, and relationships within each image to enable modeling of these interactions.

The dataset contains over 108K images, with each image averaging 35 objects, 26 attributes, and 21 pairwise relationships between objects, and the objects, attributes, relationships, and noun phrases in region descriptions and question-answer pairs are canonicalized to WordNet synsets. Together these annotations form what the authors describe as the densest and largest dataset of image descriptions, objects, attributes, relationships, and question-answer pairs, providing a resource for learning the relational understanding that cognitive visual tasks demand.