Scaling Instruction-Finetuned Language Models

Starting from the observation that finetuning language models on datasets phrased as instructions improves both performance and generalization to unseen tasks, the paper systematically studies instruction finetuning by scaling three factors: the number of finetuning tasks, the size of the model, and the inclusion of chain-of-thought reasoning data. The approach is evaluated across several model classes, including PaLM, T5, and U-PaLM, over multiple prompting setups such as zero-shot, few-shot, and chain-of-thought, and across a range of benchmarks including MMLU, BBH, TyDiQA, MGSM, and open-ended generation.

Instruction finetuning dramatically improves performance across these settings; for example, Flan-PaLM 540B finetuned on 1.8K tasks outperforms PaLM 540B by an average of 9.4% and reaches state-of-the-art results such as 75.2% on five-shot MMLU. The authors also publicly release Flan-T5 checkpoints, which deliver strong few-shot performance even compared with much larger models like PaLM 62B, establishing instruction finetuning as a general and practical method for improving the performance and usability of pretrained language models.