Dueling Network Architectures for Deep Reinforcement Learning

While many reinforcement learning successes relied on conventional architectures such as convolutional networks, LSTMs, or auto-encoders, this paper presents a new neural network architecture designed specifically for model-free reinforcement learning. The dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. This factoring generalizes learning across actions without imposing any change on the underlying reinforcement learning algorithm.

The main benefit of separating value and advantage is better policy evaluation in the presence of many similar-valued actions, a common situation in which distinguishing action quality is difficult. As a result, the dueling architecture enables the reinforcement learning agent to outperform the state of the art on the Atari 2600 domain, and it became a widely adopted building block in deep reinforcement learning.