High-Dimensional Continuous Control Using Generalized Advantage Estimation

This paper tackles two central challenges of policy gradient reinforcement learning: the large number of samples typically required and the difficulty of obtaining stable, steady improvement given nonstationary data. To reduce sample cost, it introduces generalized advantage estimation, which uses value functions and an exponentially-weighted estimator of the advantage function, analogous to TD(lambda), to substantially cut the variance of policy gradient estimates at the cost of some bias. To improve stability, it applies a trust region optimization procedure to both the policy and the value function, each represented by neural networks.

The fully model-free method produced strong empirical results on highly challenging 3D locomotion tasks, learning running gaits for simulated bipedal and quadrupedal robots and a policy for getting a biped to stand up from lying on the ground. In contrast to prior work relying on hand-crafted policy representations, the neural network policies mapped directly from raw kinematics to joint torques, with the biped learning requiring simulated experience equivalent to only one to two weeks of real time. GAE became a widely used technique for stable, sample-efficient policy gradient training.