Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

This paper proposes a combined bottom-up and top-down visual attention mechanism for image captioning and visual question answering. Whereas top-down attention alone had been used extensively for fine-grained analysis and even multiple steps of reasoning, the authors argue attention is most naturally computed at the level of objects and other salient image regions. Their bottom-up mechanism, based on Faster R-CNN, proposes candidate image regions each with an associated feature vector, and the top-down mechanism then determines the feature weightings.

Applied to image captioning, the method set a new state-of-the-art on the MSCOCO test server, achieving CIDEr, SPICE, and BLEU-4 scores of 117.9, 21.5, and 36.9. The same approach also took first place in the 2017 VQA Challenge, demonstrating the broad applicability of grounding attention in detected object and salient image regions across both captioning and visual question answering.