ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

ALBERT (A Lite BERT) addresses the fact that increasing model size in language-representation pretraining improves downstream tasks but eventually hits GPU/TPU memory limits and longer training times. It introduces two parameter-reduction techniques that lower memory consumption and increase training speed relative to the original BERT, and it uses a self-supervised loss focused on modeling inter-sentence coherence.

Comprehensive empirical evidence shows these changes let the model scale much better than BERT, and the coherence loss consistently helps downstream tasks with multi-sentence inputs. The best ALBERT configuration establishes new state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters than BERT-large, demonstrating that smarter parameter use can beat brute-force size.