End to End Learning for Self-Driving Cars

This work applies end-to-end learning to self-driving cars by training a convolutional neural network to map raw pixels from a single front-facing camera directly to steering commands. Rather than decomposing the problem into explicit stages such as lane-marking detection, path planning, and control, the system optimizes all processing steps simultaneously, using only the human driver's steering angle as the training signal. It automatically learns internal representations of necessary steps, such as detecting useful road features, without ever being explicitly trained to do so.

With minimal human training data, the system learned to drive in traffic on local roads with or without lane markings, on highways, and in challenging areas with unclear visual guidance such as parking lots and unpaved roads. The authors argue that joint end-to-end optimization should yield better performance and smaller networks than hand-designed intermediate criteria. Trained with an NVIDIA DevBox and Torch 7 and deployed on an NVIDIA DRIVE PX running at 30 frames per second, it demonstrated the surprising power of the end-to-end approach.