Papperoni

2016

Bridging Nonlinearities and Stochastic Regularizers With Gaussian Error Linear Units

Dan Hendrycks, Kevin Gimpel

citations

Cite Score

AI summary

This paper introduces the Gaussian Error Linear Unit (GELU), a novel neural network activation function inspired by stochastic regularizers like dropout and zoneout. GELU is evaluated on MNIST and CIFAR-10, achieving performance improvements over ReLU and ELU activations.

Main Contributions

Introduces the Gaussian Error Linear Unit (GELU) activation function, bridging nonlinearities and stochastic regularizers.
Proposes a new probabilistic understanding of nonlinearities based on the connection between GELU and stochastic regularizers.
Empirically evaluates GELU against ReLU and ELU on MNIST and CIFAR-10 datasets.
Demonstrates that GELU achieves performance improvements across all tasks compared to ReLU and ELU.
Shows GELU surpasses ELU and ReLU for shallow and deep convolutional neural networks

Abstract

We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU nonlinearity is the expected transformation of a stochastic regularizer which randomly applies the identity or zero map, combining the intuitions of dropout and zoneout while respecting neuron values. This connection suggests a new probabilistic understanding of nonlinearities. We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all tasks.

Citation Graph

Loading graph...

References [14]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Google Scholar

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

Google Scholar

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[3]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Google Scholar

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[4]Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2010

18 papers in library cite

Google Scholar

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

[5]Rectifier Nonlinearities Improve Neural Network Acoustic Models

A. L. Maas, A. Y. Hannun, Andrew Y. Ng - 2013

3 papers in library cite

Google Scholar

It's a good paper, but I expected SO MUCH MORE! In the end it's like "eh, leaky relus train slightly faster" - uuughhh

[6]Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

D. A. Clevert, Thomas Unterthiner, Sepp Hochreiter - 2016

2 papers in library cite

Google Scholar

Very nice paper. I am not sure how influential it is, but I like how they explain the whole idea and motivation behind the ELU. I also like that the math is contained in a single section, and they give you a final summary to help understand what the math means.

[7]Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks

Surya Ganguli - 2014

9 papers in library cite

Google Scholar

TBH it's been almost 2 months since I read this paper (shame on me for forgetting to add it)... Anyway, as I recall it I liked it, but TBH it's a bit underwhelming because it solved only for linear networks

[8]Deep Networks With Stochastic Depth

G. Huang, Y. S. Sun, Ze Liu, D. Sedra, K. Q. Weinberger - 2016

3 papers in library cite

Google Scholar

'Stochastic Depth' - randomly dropping some layers

[9]Residual Networks Behave Like Ensembles of Relatively Shallow Networks

A. Veit, M. J. Wilber, S. Belongie - 2016

4 papers in library cite

Google Scholar

Resnets are ensembles

[10]All You Need Is a Good Init

D. Mishkin, J. Matas - 2016

2 papers in library cite

Google Scholar

Importance of initialization

[11]Learning With Pseudo-Ensembles

P. Bachman, O. Alsharif, D. Precup - 2014

1 paper in library cites

Google Scholar

I liked the title, and I think they argue that NNs are ensembles

[12]Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

David Krueger, T. Maharaj, J. Kramar, M. Pezeshki, Nicolas Ballas, N. R. Ke, A. G. A. P. Goyal, Yoshua Bengio, Hugo Larochelle, Aaron Courville - 2016

3 papers in library cite

Google Scholar

Zoneout: variation of dropout, but with very little citations

Missing year

[13]Generalizing and Improving Weight Initialization

Dan Hendrycks, Kevin Gimpel

1 paper in library cites

Google Scholar

[14]A Simple Approximation to the Area Under Standard Normal Curve

A. Choudhury - 2014

1 paper in library cites

Google Scholar

Cited by

papers in your library

Cites

papers in your library

Read

on August 6, 2025

Very understandable, and very nice! I don't think the justification is good, but hey, it works!