Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

This paper introduces the exponential linear unit (ELU), a new activation function for deep neural networks. Like rectified linear units and their variants, ELUs alleviate the vanishing gradient problem through the identity for positive values, but unlike ReLUs they also take on negative values. These negative values push mean unit activations closer to zero, similar to batch normalization but with lower computational cost, which reduces the bias-shift effect and brings the normal gradient closer to the unit natural gradient. ELUs also saturate to a negative value for smaller inputs, giving a noise-robust deactivation state.

In experiments, ELUs lead to both faster learning and significantly better generalization than ReLUs and LReLUs on networks deeper than five layers. On CIFAR-100 ELU networks significantly outperform ReLU networks with batch normalization, while batch normalization does not further help ELU networks; ELU networks rank among the top reported CIFAR-10 results and give the best published CIFAR-100 result without multi-view evaluation or model averaging. On ImageNet they considerably speed up learning versus an equivalent ReLU network, reaching under 10% classification error for a single-crop, single model.