Prefix-Tuning: Optimizing Continuous Prompts for Generation

Prefix-tuning is a lightweight alternative to full fine-tuning for natural language generation. Rather than updating all of a large pretrained language model's parameters, which forces storing a separate full copy for every task, prefix-tuning keeps the model frozen and optimizes a short sequence of continuous, task-specific vectors called the prefix. Inspired by prompting, the method lets subsequent tokens attend to this prefix as if it were a set of virtual tokens. The authors apply it to GPT-2 for table-to-text generation and BART for summarization.

By learning only about 0.1% of the parameters, prefix-tuning obtains performance comparable to full fine-tuning in the full-data setting, outperforms fine-tuning in low-data settings, and extrapolates better to examples with topics unseen during training. This mattered because it offered a far more parameter-efficient and modular way to adapt large language models to many tasks without duplicating the entire model.