Prioritized Experience Replay

Experience replay allows online reinforcement learning agents to store and reuse past transitions, but prior methods sampled uniformly from the replay memory, replaying transitions at the same frequency they were originally experienced regardless of how significant they were. The authors develop a framework for prioritizing experience so that important transitions are replayed more frequently, letting the agent learn more efficiently from the transitions that matter most.

They apply prioritized experience replay to Deep Q-Networks, the algorithm that reached human-level performance across many Atari games. DQN with prioritized replay achieves a new state of the art, outperforming DQN with uniform replay on 41 out of 49 games, demonstrating that how experiences are sampled, not just which experiences are stored, is central to efficient deep reinforcement learning.