Training language models to follow instructions with human feedback

This paper shows an avenue for aligning language models with user intent across a wide range of tasks by fine-tuning with human feedback, addressing the observation that making models bigger does not inherently make them better at following intent. Starting from labeler-written prompts and prompts submitted through the OpenAI API, the authors collect labeler demonstrations of desired behavior to fine-tune GPT-3 with supervised learning, then collect rankings of model outputs and further fine-tune the supervised model using reinforcement learning from human feedback, producing the InstructGPT models.

In human evaluations on the paper's prompt distribution, outputs from the 1.3B-parameter InstructGPT model are preferred to outputs from the 175B-parameter GPT-3 despite having 100x fewer parameters, and InstructGPT shows improvements in truthfulness and reductions in toxic output generation with minimal performance regressions on public NLP datasets. Although InstructGPT still makes simple mistakes, the results establish fine-tuning with human feedback as a promising direction for aligning language models with human intent.