End-to-End Training of Deep Visuomotor Policies

The paper investigates whether jointly training perception and control systems end-to-end yields better performance than training each component separately, a question motivated by the hand-engineered perception, state estimation, and low-level control components common in practical policy search. The authors develop a method that learns policies mapping raw image observations directly to torques at the robot's motors, using deep convolutional neural networks with 92,000 parameters. Training relies on a partially observed guided policy search method that transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method.

The method is evaluated on a range of real-world manipulation tasks that demand close coordination between vision and control, such as screwing a cap onto a bottle, and it includes simulated comparisons against a range of prior policy search methods. This work mattered because it demonstrated that perception and control could be trained together end-to-end from raw pixels to torques on real robots, moving beyond pipelines of separately engineered modules.