Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

The paper applies preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models so that they act as helpful and harmless assistants. A central element of the method is an iterated online mode of training, in which the preference models and the reinforcement learning policies are updated on a roughly weekly cadence using fresh human feedback data, which efficiently improves both the datasets and the resulting models over successive rounds.

The authors find that this alignment training improves performance on almost all NLP evaluations and is fully compatible with training for specialized skills such as Python coding and summarization, so alignment does not come at the expense of capability. Investigating the robustness of RLHF, they identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization, and they provide additional analyses of calibration, competing objectives, and out-of-distribution detection along with comparisons to human writers.