HellaSwag: Can a Machine Really Finish Your Sentence?

HellaSwag targets commonsense natural language inference, where a system reads an event description and selects the most likely follow-up sentence. The authors construct it using Adversarial Filtering, a data-collection method in which a series of discriminators iteratively pick an adversarial set of machine-generated wrong answers. The key design choice is to scale the length and complexity of examples into a 'Goldilocks' zone where generated text looks absurd to people but frequently fools strong models.

Although the questions are trivial for humans, who exceed 95% accuracy, state-of-the-art models score below 48%, showing that commonsense inference remained unsolved even after BERT approached human-level performance on earlier tasks. The construction and its resulting difficulty reveal limitations in deep pretrained models and point toward a research path where benchmarks co-evolve adversarially with the evolving state of the art to keep posing ever-harder challenges.