Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

Grounding DINO is an open-set object detector created by marrying the Transformer-based detector DINO with grounded pre-training, so it can detect arbitrary objects from human inputs such as category names or referring expressions. The key idea for open-set detection is to introduce language into a closed-set detector to enable open-set concept generalization. To fuse language and vision effectively, the authors conceptually divide a closed-set detector into three phases and propose a tight fusion solution comprising a feature enhancer, language-guided query selection, and a cross-modality decoder.

Unlike prior work that mainly evaluated open-set detection on novel categories, the authors also evaluate on referring expression comprehension for objects specified with attributes, testing across COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO performs strongly in all settings, achieving 52.5 AP on the COCO zero-shot transfer benchmark without any COCO training data and setting a new record of 26.1 mean AP on the ODinW zero-shot benchmark, demonstrating robust open-vocabulary detection.