Deep Reinforcement Learning from Human Preferences

The paper explores how to communicate complex goals to reinforcement learning systems by defining those goals in terms of non-expert human preferences between pairs of trajectory segments, rather than a hand-specified reward function. A human observer compares short segments of agent behavior, and these preference judgments are used to guide learning, allowing the agent to pursue objectives that are hard to encode directly as rewards.

The approach effectively solves complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while requiring feedback on less than one percent of the agent's interactions with the environment. This dramatically reduces the cost of human oversight, making it practical to apply to state-of-the-art RL systems, and the authors further show they can train novel complex behaviors with about an hour of human time.