On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

Reflecting on three years of NLP dominated by ever-larger language models for English, including BERT and its variants, GPT-2/3, and Switch-C, the authors note these models advanced benchmark state of the art through both architectural innovation and sheer scale, typically via pretraining and task-specific fine-tuning. Rather than propose a new model, the paper takes a step back to ask how big is too big, examining the possible risks associated with this trajectory and the paths available for mitigating them.

The authors offer concrete recommendations: weigh the environmental and financial costs of large models first, invest resources in carefully curating and documenting datasets instead of ingesting everything on the web, carry out pre-development exercises that evaluate how a planned approach fits research and development goals and supports stakeholder values, and encourage research directions that go beyond building ever-larger language models. The paper frames these as ways to pursue NLP progress more responsibly rather than defaulting to scale.