Show and tell: A neural image caption generator

The paper tackles automatic image captioning, a fundamental problem connecting computer vision and natural language processing. It presents a single generative model built on a deep recurrent architecture that combines recent advances in computer vision and machine translation to generate natural-language sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image, learning to produce fluent language solely from image descriptions.

Experiments on several datasets show the model is accurate and fluent, verified both qualitatively and quantitatively. It raises the BLEU-1 score on the Pascal dataset from the prior state of the art of 25 to 59, approaching human performance of around 69, and improves BLEU-1 on Flickr30k from 56 to 66 and on SBU from 19 to 28. On the newly released COCO dataset it achieves a BLEU-4 of 27.7, the state of the art at the time, demonstrating the strength of the neural encoder-decoder approach to captioning.