Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

This paper addresses a weakness in visual question answering: because structure in the world and bias in language are easier signals to learn than visual content, VQA models often ignore the image and still answer well, inflating their apparent ability. To make vision matter, the authors rebalance the popular VQA dataset so that every question is paired not with a single image but with two similar images that lead to two different answers. The resulting VQA v2.0 dataset is more balanced by construction, contains roughly twice as many image-question pairs, and is released publicly.

Benchmarking several state-of-the-art VQA models on the balanced dataset, the authors find that all of them perform significantly worse than before, providing the first concrete empirical evidence that these models exploit language priors rather than genuinely understanding images. The work also reports insights from the 2017 VQA Challenge run on the new data, and its complementary-image collection protocol enables a novel interpretable model that justifies its answer by pointing to a similar image it believes has a different answer, helping build user trust.