Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

The paper presents the AI2 Reasoning Challenge (ARC), a new benchmark for advanced question answering built from natural, grade-school science questions written for human tests. The dataset is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions that both a retrieval-based algorithm and a word co-occurrence algorithm answer incorrectly. Alongside the 7,787-question set, the authors release the ARC Corpus of 14 million science sentences and implementations of three neural baseline models.

With 7,787 questions, ARC is the largest public-domain set of its kind, and the authors test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks. None significantly outperform a random baseline, underscoring that the challenge demands far more powerful knowledge and reasoning than prior datasets. The authors pose ARC to the community as an open challenge to spur progress in question answering.