CIDEr: Consensus-based image description evaluation

The paper tackles the challenge of automatically evaluating sentence-level image descriptions, a problem that grew in importance as object detection, attribute classification, and action recognition improved caption generation. It introduces a paradigm built on human consensus, comprising a new triplet-based method for collecting human annotations that measure agreement, a new automated metric (CIDEr) designed to capture that consensus, and two purpose-built datasets, PASCAL-50S and ABSTRACT-50S, which supply 50 reference sentences for each image.

Across sentences produced by varied sources, the CIDEr metric captures human judgments of consensus more faithfully than existing metrics, and the authors use it to evaluate five state-of-the-art image description systems, establishing a benchmark for future comparison. The approach's practical impact is reflected in CIDEr-D, a variant integrated into the MS COCO evaluation server to support systematic, standardized benchmarking of caption generation.