Addressing Function Approximation Error in Actor-Critic Methods

The paper investigates how function approximation errors, known in value-based methods such as deep Q-learning to cause overestimated value estimates and suboptimal policies, also persist in the actor-critic setting. It proposes novel mechanisms to minimize these effects on both the actor and the critic. Building on Double Q-learning, the algorithm takes the minimum value between a pair of critics to limit overestimation, and it draws a connection between target networks and overestimation bias.

The authors further suggest delaying policy updates relative to value updates to reduce per-update error and further improve performance. Evaluated on the suite of OpenAI Gym tasks, the method outperforms the state of the art in every environment tested. This mattered because the targeted fixes to overestimation bias delivered consistent, state-of-the-art performance across all tested continuous-control environments.