Natural Questions: A Benchmark for Question Answering Research

Natural Questions is a question answering data set whose questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question and a Wikipedia page from the top 5 search results and annotates a long answer, typically a paragraph, and a short answer of one or more entities if present, or marks null when no long or short answer appears on the page. The public release consists of 307,373 training examples with single annotations, 7,830 development examples with 5-way annotations, and 7,842 5-way annotated examples sequestered as test data.

The authors present experiments validating the quality of the data and analyze 25-way annotations on 302 examples to give insights into human variability on the annotation task. They introduce robust metrics for evaluating question answering systems, demonstrate high human upper bounds on these metrics, and establish baseline results using competitive methods drawn from related literature, providing a realistic benchmark for open-domain question answering.