Universal Language Model Fine-tuning for Text Classification

This paper introduces Universal Language Model Fine-tuning (ULMFiT), a transfer-learning method for natural language processing. The authors note that while inductive transfer learning had greatly impacted computer vision, existing NLP approaches still required task-specific modifications and training from scratch. ULMFiT is designed to be applied to any NLP task, and the paper introduces techniques that are key for effectively fine-tuning a language model without those task-specific changes.

The method significantly outperforms the state of the art on six text-classification tasks, reducing error by 18 to 24 percent on the majority of the datasets. It is also strikingly data-efficient: with only 100 labeled examples, ULMFiT matches the performance of training from scratch on 100 times as much data. The authors open-source their pretrained models and code, providing a single fine-tuning method that transfers to any NLP task without task-specific architecture changes or training from scratch.