Gaussian Error Linear Units (GELUs)

The paper proposes the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function defined as xΦ(x), where Φ(x) is the standard Gaussian cumulative distribution function. The key idea is that GELU weights its inputs by their value, in contrast to the ReLU nonlinearity, which gates inputs by their sign by multiplying x with the indicator that x is greater than zero.

The authors perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activation functions. They find performance improvements across all considered tasks, spanning computer vision, natural language processing, and speech. This mattered because it provided a simple, high-performing activation that consistently improved on the standard ReLU and ELU choices across multiple modalities.