Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

This paper studies rectifier neural networks for image classification from two aspects. First, it proposes the Parametric Rectified Linear Unit (PReLU), a generalization of the traditional rectified unit that improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, it derives a robust initialization method that explicitly considers rectifier nonlinearities, enabling extremely deep rectified models to be trained directly from scratch and allowing investigation of deeper or wider network architectures.

Based on the learnable activation and advanced initialization, the models achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset — a 26% relative improvement over the ILSVRC 2014 winner GoogLeNet (6.66%). To the authors' knowledge this was the first result to surpass the reported human-level performance of 5.1% on this dataset.