Parameter-Efficient Transfer Learning for NLP

The paper addresses the parameter inefficiency of fine-tuning large pre-trained models when many downstream NLP tasks are involved, since standard fine-tuning produces an entire new model per task. As an alternative, it proposes transfer learning with adapter modules: small trainable components inserted into the network that add only a few parameters per task while the original model parameters remain fixed, producing a compact and extensible model with a high degree of parameter sharing and allowing new tasks to be added without revisiting previous ones.

To demonstrate effectiveness, the authors transfer the BERT Transformer to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, coming within 0.4% of full fine-tuning on GLUE while adding only 3.6% of parameters per task, compared with the 100% of parameters trained by fine-tuning. This showed that strong transfer performance is achievable with far greater parameter efficiency, which matters when serving many tasks.