Finetuned Language Models Are Zero-Shot Learners

This paper explores instruction tuning as a simple method for improving the zero-shot learning abilities of language models. The authors take a 137-billion-parameter pretrained language model and finetune it on more than 60 NLP tasks that are each verbalized through natural language instruction templates, producing a model they call FLAN. The idea is that teaching the model to follow instructions across many tasks should help it generalize to new tasks described the same way, without task-specific examples.

Evaluated on unseen task types, FLAN substantially improves over its unmodified pretrained counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 evaluated tasks, even outperforming few-shot GPT-3 by a large margin on benchmarks such as ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that the number of finetuning datasets, the scale of the model, and the use of natural language instructions are all key to the success of instruction tuning. The work helped establish instruction tuning as a core technique for aligning and generalizing large language models.