RoBERTa: A Robustly Optimized BERT Pretraining Approach

This paper conducts a careful replication study of BERT pretraining, motivated by the difficulty of fairly comparing language model pretraining approaches given expensive training, private datasets of varying size, and hyperparameter choices that significantly affect final results. The core method systematically measures the impact of key hyperparameters and the amount of training data on BERT's pretraining, rather than proposing a new architecture.

The central finding is that BERT was significantly undertrained in its original release, and that a version retrained with better-tuned choices and more data can match or exceed the performance of every model published after it, achieving state-of-the-art results on GLUE, RACE, and SQuAD. This mattered because it showed that much of the reported improvement attributed to newer architectures may instead stem from previously overlooked pretraining design choices, raising questions about the true source of progress, and the authors released their models and code to support further research.