Deep visual-semantic alignments for generating image descriptions

This paper presents a model that generates natural language descriptions of images and of specific regions within them. The approach leverages datasets of images paired with sentence descriptions to learn the inter-modal correspondences between language and visual data. Its alignment model rests on a novel combination of convolutional neural networks over image regions, bidirectional recurrent neural networks over sentences, and a structured objective that aligns the two modalities in a shared multimodal embedding. A Multimodal Recurrent Neural Network then uses the inferred region-word alignments to learn to generate novel descriptions of image regions.

The alignment model achieves state-of-the-art results in retrieval experiments on the Flickr8K, Flickr30K, and MSCOCO datasets. The descriptions produced by the generative model significantly outperform retrieval-based baselines, both on full images and on a new dataset of region-level annotations. By connecting vision and language at the level of regions, the work advanced image captioning and dense description generation.