The Power of Scale for Parameter-Efficient Prompt Tuning

The paper introduces prompt tuning, a simple yet effective mechanism for learning soft prompts that condition a frozen language model to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, these soft prompts are learned through backpropagation and can incorporate signals from any number of labeled examples. The authors position the method as a simplification of the prefix tuning of Li and Liang (2021) and compare it to that and other similar approaches.

The end-to-end learned approach outperforms GPT-3's few-shot learning by a large margin, and ablations on model size using T5 show that prompt tuning becomes more competitive as scale increases: once models exceed billions of parameters, it closes the gap and matches the strong performance of full model tuning. This matters because large models are costly to share and serve, so being able to reuse a single frozen model for many downstream tasks eases that burden. Conditioning a frozen model with soft prompts also confers robustness to domain transfer and enables efficient prompt ensembling, and the authors release code and checkpoints.