Papperoni

2010

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network With a Local Denoising Criterion

P. H. Vincent, Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre Antoine Manzagol

Open PDF Google Scholar

citations

Cite Score

84

AI summary

This paper introduces Stacked Denoising Autoencoders (SDAE) that learn robust representations by denoising corrupted inputs, demonstrating superior classification performance on benchmarks like MNIST and various image datasets, outperforming DBNs and SAEs.

Main Contributions

Introduced Stacked Denoising Autoencoders (SDAE) for deep network pretraining using a local denoising criterion.
Demonstrated that SDAE significantly lowers classification error on benchmark problems, outperforming or matching Deep Belief Networks (DBNs) and Stacked Autoencoders (SAEs).
Showed that higher-level representations learned by SDAE boost the performance of subsequent SVM classifiers.
Qualitative experiments revealed that denoising autoencoders learn useful features like Gabor-like edge detectors from natural images and stroke detectors from digit images, unlike ordinary autoencoders.
Established denoising as a tractable unsupervised objective for learning useful higher-level representations, addressing limitations of traditional autoencoders in learning over-complete representations.

Abstract

We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations.

Citation Graph

Loading graph...

References [53]

Sort:

Filter:

[1]Neural Networks and Physical Systems With Emergent Collective Computational Abilities

J. J. Hopfield - 1982

8 papers in library cite

Very theory heavy and doesn't seem to add much... Famous because it introduced the "Hopfield network" - "At a time when research in neural networks was facing a lull, Hopfield's paper injected a fresh perspective by demonstrating how a network of simple, interconnected units could exhibit complex computational abilities as a collective, emergent property."

[2]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[3]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[4]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[5]Learning Deep Architectures for AI

Yoshua Bengio - 2009

25 papers in library cite

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

[6]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[7]Emergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images

B. Olshausen, D. Field - 1996

5 papers in library cite

I can see the contribution, but overall the paper was uninteresting

[8]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[9]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Good read, but I think I need to revisit it after I understand RBMs better.

[10]Why Does Unsupervised Pre-Training Help Deep Learning?

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre Antoine Manzagol, Pascal Vincent, Samy Bengio - 2010

12 papers in library cite

Good paper, easy to follow, and brings some light to the pre-training stuff (layer-by-layer). I just wish it wasn't so long. It's a chore.

[11]Connectionist Learning Procedures

Geoffrey E. Hinton - 1987

11 papers in library cite

It's a very good overview of everything that was happening in 1987! A bit too long though, but a good start nonetheless.

[12]Scaling Learning Algorithms Towards AI

Yoshua Bengio, Yann Lecun - 2007

15 papers in library cite

I should have read this sooner! Such a good explanation of why deep learning > other stuff! Also, better than Bengio's 2006 Learning Deep Archs for AI

[13]Efficient Learning of Sparse Representations With an Energy-Based Model

Marc'aurelio Ranzato, C. Poultney, S. Chopra, Yann Lecun - 2006

20 papers in library cite

It's ok. Not really good, but alright.

[14]An Empirical Evaluation of Deep Architectures on Problems With Many Factors of Variation

Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio - 2007

13 papers in library cite

Good paper showing promising results for Deep Learning. Nothing amazing but good nonetheless

[15]Deep Learning via Semi-Supervised Embedding

Jason Weston, F. Ratle, Ronan Collobert - 2008

10 papers in library cite

It's a good paper and nice idea, but seems overly complicated and I don't think it's very used... (PS: this was republished in 2012)

[16]Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1

D. Rumelhart, J. Mcclelland, T. P. R. Group - 1986

5 papers in library cite

Book, 500+ pages

[17]Information Processing in Dynamical Systems: Foundations of Harmony Theory

P. Smolensky - 1986

11 papers in library cite

88 pages; Introduced RBMs

[18]Sparse Feature Learning for Deep Belief Networks

Marc'aurelio Ranzato, Y. Boureau, Yann Lecun - 2008

12 papers in library cite

[19]Sparse Coding With an Overcomplete Basis Set: A Strategy Employed by V1?

Bruno A. Olshausen, David J. Field - 1997

10 papers in library cite

[20]Sparse Deep Belief Net Model for Visual Area V2

Honglak Lee, C. Ekanadham, A. Ng - 2008

10 papers in library cite

[21]Modèles Connexionnistes De l'Apprentissage

Yann Lecun - 1987

9 papers in library cite

[22]Exploring Strategies for Training Deep Neural Networks

Hugo Larochelle, Yoshua Bengio, J. Louradour, P. Lamblin - 2009

7 papers in library cite

40 pages

[23]On the Power of Small-Depth Threshold Circuits

J. Hastad, M. Goldmann - 1991

7 papers in library cite

[24]Receptive Fields of Single Neurons in the Cat's Striate Cortex

D. H. Hubel, T. N. Wiesel - 1959

6 papers in library cite

[25]Justifying and Generalizing Contrastive Divergence

Yoshua Bengio, O. Delalleau - 2007

5 papers in library cite

[26]Learning Continuous Attractors in Recurrent Networks

S. H. Seung - 1998

5 papers in library cite

[27]Many-Layered Learning

P. Utgoff, D. Stracuzzi - 2002

5 papers in library cite

[28]Semi-Supervised Learning

O. Chapelle, B. Scholkopf, A. Zien - 2006

5 papers in library cite

[29]Almost Optimal Lower Bounds for Small Depth Circuits

J. Hastad - 1986

4 papers in library cite

[30]Auto-Association by Multilayer Perceptrons and Singular Value Decomposition

H. Bourlard, Y. Kamp - 1988

4 papers in library cite

[31]Tangent Prop-A Formalism for Specifying Selected Invariances in an Adaptive Neural Network

Patrice Simard, B. Victorri, Yann Lecun, John Denker - 1992

4 papers in library cite

[32]The "Independent Components" of Natural Scenes Are Edge Filters

A. Bell, T. Sejnowski - 1997

4 papers in library cite

[33]Dependency Networks for Inference, Collaborative Filtering, and Data Visualization

D. Heckerman, D. M. Chickering, C. Meek, R. Rounthwaite, C. Kadie - 2000

3 papers in library cite

[34]Memoires Associatives Distribuees

P. Gallinari, Yann Lecun, S. Thiria, F. F. Soulie - 1987

3 papers in library cite

[35]Natural Image Denoising With Convolutional Networks

V. Jain, S. Seung - 2008

3 papers in library cite

[36]Neural Networks and Principal Component Analysis: Learning From Examples Without Local Minima

P. Baldi, Kur Hornik - 1989

3 papers in library cite

[37]An Information Maximisation Approach to Blind Separation and Blind Deconvolution

A. Bell, T. Sejnowski - 1995

2 papers in library cite

[38]Document Image Defect Models

H. Baird - 1990

2 papers in library cite

[39]Nonlinear Autoassociation Is Not Equivalent to PCA

N. Japkowicz, S. J. Hanson, M. A. Gluck - 2000

2 papers in library cite

[40]Statistical Analysis of Non-Lattice Data

J. Besag - 1975

2 papers in library cite

[41]Training With Noise Is Equivalent to Tikhonov Regularization

C. M. Bishop - 1995

2 papers in library cite

[42]Using Additive Noise in Back-Propagation Training

L. Holmstrom, P. Koistinen - 1992

2 papers in library cite

[43]Algorithms for Classifying Recorded Music by Genre

James Bergstra - 2006

1 paper in library cites

[44]An Application of the Principle of Maximum Information Preservation to linear Systems

R. Linsker - 1989

1 paper in library cites

[45]Creating Artificial Neural Networks That Generalize

J. Sietsma, R. Dow - 1991

1 paper in library cites

[46]Deep Learning Using Robust Interdependent Codes

Hugo Larochelle, Dumitru Erhan, Pascal Vincent - 2009

1 paper in library cites

[47]Emergence of Grandmother Memory in Feed Forward Networks: Learning With Noise and Forgetfulness

R. Scalettar, A. Zee - 1988

1 paper in library cites

[48]Factors Influencing Learning by Back-Propagation

A. V. Lehman, E. G. Paek, P. F. Liao, A. Marrakchi, J. S. Patel - 1988

1 paper in library cites

[49]Incorporating Invariances in Support Vector Learning Machines

B. Scholkopf, C. J. C. Burges, V. Vapnik - 1996

1 paper in library cites

[50]Kernel Methods for Deep Learning

Y. Cho, L. Saul - 2010

1 paper in library cites

[51]Noise Injection: Theoretical Prospects

Y. Grandvalet, S. Canu, S. Boucheron - 1997

1 paper in library cites

[52]Recognition and Structure From One 2d Model View: Observations on Prototypes, Object Classes and Symmetries

T. Poggio, T. Vetter - 1992

1 paper in library cites

[53]The Effects of Adding Noise During Backpropagation Training on a Generalization Performance

G. An - 1996

1 paper in library cites

Cited by

6

papers in your library

Cites

18

papers in your library

Read

on February 17, 2026

This is basically a summary of everything that happened from 2006-2010, and also points some interesting things about DBNs! Very well explained as well.

Tags

Paper Aliases

No aliases