Generative Adversarial Imitation Learning

This paper addresses imitation learning: recovering a policy purely from examples of expert behavior, without interacting with the expert and without any reinforcement (reward) signal. A common existing strategy is indirect, first using inverse reinforcement learning to recover the expert's cost function and then running reinforcement learning on that cost to obtain a policy, a pipeline that can be slow. The authors instead propose a general framework for directly extracting a policy from data, behaving as though it had been produced by reinforcement learning applied after inverse reinforcement learning.

A particular instantiation of this framework reveals a close analogy between imitation learning and generative adversarial networks, and from it the authors derive a model-free imitation learning algorithm known as Generative Adversarial Imitation Learning. The method obtains significant performance gains over existing model-free approaches when imitating complex behaviors in large, high-dimensional environments. This GAN-based formulation became a foundational technique for scalable imitation learning.