Papperoni

2010

Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton

Open PDF Google Scholar

citations

Cite Score

94

AI summary

This paper introduces noisy rectified linear units (NReLUs) to improve Restricted Boltzmann Machines (RBMs). NReLUs learn features better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset, preserving information about relative intensities through multiple layers of feature detectors.

Main Contributions

Introduces noisy rectified linear units (NReLUs) as an alternative to binary units in Restricted Boltzmann Machines (RBMs).
Demonstrates that NReLUs learn features that are better for object recognition compared to binary units.
Shows that NReLUs preserve information about relative intensities, which is beneficial for tasks involving images with varying lighting conditions.
Presents empirical results on the NORB dataset and the Labeled Faces in the Wild dataset, showing improved performance with NReLUs.
Provides an approximate probabilistic interpretation for the max(0, x) nonlinearity.

Abstract

Restricted Boltzmann machines were developed using binary stochastic hidden units. These can be generalized by replacing each binary unit by an infinite number of copies that all have the same weights but have progressively more negative biases. The learning and inference rules for these "Stepped Sigmoid Units" are unchanged. They can be approximated efficiently by noisy, rectified linear units. Compared with binary units, these units learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset. Unlike binary units, rectified linear units preserve information about relative intensities as information travels through multiple layers of feature detectors.

Citation Graph

Loading graph...

References [21]

Sort:

Filter:

[1]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[2]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[3]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Good read, but I think I need to revisit it after I understand RBMs better.

[4]What Is the Best Multi-Stage Architecture for Object Recognition?

K. Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun - 2009

20 papers in library cite

So boring and convoluted... They try to make things more abstract by saying "filter banks" and "2 stages", but in the end it's CNNs. It seems like they are trying to tie NNs to the rest of ML, but it doesn't work.

[5]Scaling Learning Algorithms Towards AI

Yoshua Bengio, Yann Lecun - 2007

15 papers in library cite

I should have read this sooner! Such a good explanation of why deep learning > other stuff! Also, better than Bengio's 2006 Learning Deep Archs for AI

[6]Learning Methods for Generic Object Recognition With Invariance to Pose and Lighting

Yann Lecun, Fu Jie Huang, Leon Bottou - 2004

18 papers in library cite

Good paper, nice methodology for creating different images. However, I think that this was not too impactful... I don't see this being used a lot.

[7]An Empirical Evaluation of Deep Architectures on Problems With Many Factors of Variation

Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio - 2007

13 papers in library cite

Good paper showing promising results for Deep Learning. Nothing amazing but good nonetheless

[8]Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments

G. B. Huang, M. Ramesh, T. Berg, E. L. Miller - 2007

5 papers in library cite

Seems relevant but I am tired of reading papers about datasets, especially the early ones.

[9]Unsupervised Learning of Distributions on Binary Vectors Using Two Layer Networks

Y. Freund, D. Haussler - 1992

8 papers in library cite

[10]Restricted Boltzmann Machines for Collaborative Filtering

Ruslan Salakhutdinov, A. Mnih, Geoffrey E. Hinton - 2007

5 papers in library cite

[11]Rate-Coded Restricted Boltzmann Machines for Face Recognition

Y. Teh, Geoffrey Hinton - 2001

4 papers in library cite

[12]Modeling Human Motion Using Binary Latent Variables

Graham W. Taylor, Geoffrey E. Hinton, S. T. Roweis - 2007

3 papers in library cite

[13]A Hierarchical Community of Experts

Geoffrey E. Hinton, B. Sallans, Zoubin Ghahramani - 1999

2 papers in library cite

[14]Attribute and Simile Classifiers for Face Verification

N. Kumar, A. C. Berg, P. N. Belhumeur, S. K. Nayar - 2009

2 papers in library cite

[15]Diffusion Networks, Product of Experts, and Factor Analysis

T. K. Marks, J. R. Movellan - 2001

2 papers in library cite

[16]Learning a Similarity Metric Discriminatively, With Application to Face Verification

S. Chopra, Raia Hadsell, Yann Lecun - 2005

2 papers in library cite

[17]Implicit Mixtures of Restricted boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2008

1 paper in library cites

[18]Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks

R. H. R. Hahnloser, H. S. Seung, J. J. Slotine - 2003

1 paper in library cites

[19]Phone Recognition Using Restricted boltzmann Machines

A. Mohamed, Geoffrey E. Hinton - 2010

1 paper in library cites

[20]Replicated Softmax: An Undirected Topic Model

Ruslan Salakhutdinov, Geoffrey E. Hinton - 2009

1 paper in library cites

[21]Similarity Scores Based on Background Samples

Lior Wolf, T. Hassner, Y. Taigman - 2009

1 paper in library cites

Cited by

18

papers in your library

Cites

7

papers in your library

Read

on June 30, 2025

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

Tags

Paper Aliases

No aliases