Papperoni

2009

Learning Deep Architectures for AI

Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

87

AI summary

This paper explores the motivations and principles behind learning algorithms for deep architectures, focusing on Deep Belief Networks and Restricted Boltzmann Machines, highlighting their success in tackling complex AI tasks by modeling high-level abstractions and distributed representations.

Main Contributions

Motivates the need for deep architectures for AI-level tasks due to their ability to represent high-level abstractions and complex functions.
Discusses the theoretical limitations of shallow architectures, suggesting that deep architectures can compactly represent highly-varying functions.
Introduces and analyzes learning algorithms for deep architectures, with a focus on Deep Belief Networks (DBNs) and Restricted Boltzmann Machines (RBMs).
Explains how unsupervised learning of single-layer models (like RBMs) can be used as building blocks for training deeper models.
Addresses the optimization challenges in training deep architectures and proposes continuation methods and greedy layer-wise training as solutions.

Abstract

Theoretical results strongly suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one needs deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers or in complicated propositional formulae re-using many sub-formulae. Searching the parameter space of deep architectures is a difficult optimization task, but learning algorithms such as those for Deep Belief Networks have recently been proposed to tackle this problem with notable success, beating the state-of-the-art in certain areas. This paper discusses the motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer models such as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks.

Citation Graph

Loading graph...

References [134]

Sort:

Filter:

[1]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[2]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[3]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[4]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[5]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[6]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[7]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[8]Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position

Kunihiko Fukushima - 1980

8 papers in library cite

Good read! First conv net? :)

[9]Efficient Backprop

Yann Lecun, Leon Bottou, G. B. Orr, Klaus Robert Muller - 1998

20 papers in library cite

The first half is very very good. The remainder is very boring.

[10]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[11]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Good read, but I think I need to revisit it after I understand RBMs better.

[12]Learning and Development in Neural Networks: The Importance of Starting Small

Jeffrey L. Elman - 1993

5 papers in library cite

This is such a nice paper! Maybe because it's written for a specific public, but it's such an easy read and ties back a lot with neural/biological concepts!

[13]Autoencoders, Minimum Description Length, and Helmholtz Free Energy

Geoffrey E. Hinton, Richard S. Zemel - 1994

3 papers in library cite

VERY MATHY. Doesn't explain anything. Overall very hard to understand.

[14]Scaling Learning Algorithms Towards AI

Yoshua Bengio, Yann Lecun - 2007

15 papers in library cite

I should have read this sooner! Such a good explanation of why deep learning > other stuff! Also, better than Bengio's 2006 Learning Deep Archs for AI

[15]Efficient Learning of Sparse Representations With an Energy-Based Model

Marc'aurelio Ranzato, C. Poultney, S. Chopra, Yann Lecun - 2006

20 papers in library cite

It's ok. Not really good, but alright.

[16]Learning Distributed Representations of Concepts

Geoffrey E. Hinton - 1986

13 papers in library cite

 Probably seminal, but a bit boring overall

[17]Unsupervised Learning of Invariant Feature Hierarchies With Applications to Object Recognition

Marc'aurelio Ranzato, F. Huang, Y. Boureau, Yann Lecun - 2007

8 papers in library cite

I thought that I had learned pretty much everything before 2015, but this paper proved me wrong. Amazing read and very nice methodology. Very simple as well!

[18]The Wake-Sleep Algorithm for Unsupervised Neural Networks

Geoffrey Hinton, Peter Dayan, B. Frey, R. Neal - 1995

9 papers in library cite

It's okay... I get the feeling that this is early autoencoders work, but the term still didn't exist. I don't think it adds nothing new though.

[19]An Empirical Evaluation of Deep Architectures on Problems With Many Factors of Variation

Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio - 2007

13 papers in library cite

Good paper showing promising results for Deep Learning. Nothing amazing but good nonetheless

[20]Three New Graphical Models for Statistical Language Modelling

A. Mnih, Geoffrey Hinton - 2007

12 papers in library cite

I don't know why this is so impactful. I didn't like it and I think this was overly complex.

[21]Products of Experts

Geoffrey E. Hinton - 1999

4 papers in library cite

Very interesting concept. This is what paved the way to RBMs and DBNs in the future.

[22]To Recognize Shapes, First Learn to Generate Images

Geoffrey Hinton - 2006

5 papers in library cite

Maybe the best explanation of deep belief nets and RBMs by Hinton.

[23]Connectionist Language Modeling for Large Vocabulary Continuous Speech Recognition

Holger Schwenk, Jean Luc Gauvain - 2002

14 papers in library cite

Only real relevance is being early. Otherwise not much to see.

[24]Efficient Training of Large Neural Networks for Language Modeling

Holger Schwenk - 2004

6 papers in library cite

Very early paper, maybe that's why it's relevant. But very uninteresting despite the nice name ("large NNs" had a very different meaning back then)

[25]Training Connectionist Models for the Structured Language Model

P. Xu, A. Emami, Frederick Jelinek - 2003

3 papers in library cite

A copy of the other paper. Same comment: I dont know what SLM is and it just sounds like a bad idea.

[26]Reinforcement Learning: An Introduction

Richard S. Sutton, A. Barto - 1998

5 papers in library cite

Book

[27]Support Vector Networks

C. Cortes, V. Vapnik - 1995

3 papers in library cite

Tons of citations, SVMs?

[28]Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1

D. Rumelhart, J. Mcclelland, T. P. R. Group - 1986

5 papers in library cite

Book, 500+ pages

[29]Indexing by Latent Semantic Analysis

S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, R. Harshman - 1990

12 papers in library cite

LSA paper

[30]A Training Algorithm for Optimal Margin Classifiers

B. E. Boser, I. Guyon, V. N. Vapnik - 1992

3 papers in library cite

SVMs?

[31]Information Processing in Dynamical Systems: Foundations of Harmony Theory

P. Smolensky - 1986

11 papers in library cite

88 pages; Introduced RBMs

[32]Learning and Relearning in Boltzmann Machines

Geoffrey E. Hinton, T. J. Sejnowski - 1986

9 papers in library cite

37 pages; Introduced Boltzmann machines

[33]Recursive Distributed Representations

J. B. Pollack - 1990

7 papers in library cite

[34]Sparse Feature Learning for Deep Belief Networks

Marc'aurelio Ranzato, Y. Boureau, Yann Lecun - 2008

12 papers in library cite

[35]Multitask Connectionist Learning

Rich Caruana - 1994

3 papers in library cite

I couldn't find it anywhere

[36]A Theoretical Analysis of Robust Coding Over Noisy Overcomplete Channels

E. Doi, D. C. Balcan, M. S. Lewicki - 2006

5 papers in library cite

only 33 citations

[37]Untersuchungen zu dynamischen neuronalen netzen

Sepp Hochreiter - 1991

18 papers in library cite

[38]Sparse Coding With an Overcomplete Basis Set: A Strategy Employed by V1?

Bruno A. Olshausen, David J. Field - 1997

10 papers in library cite

[39]The Nature of Statistical Learning Theory

V. Vapnik - 1995

9 papers in library cite

Book, 300+ pages

[40]Exponential Family Harmoniums With an Application to Information Retrieval

M. Welling, M. R. Zvi, Geoffrey Hinton - 2005

8 papers in library cite

[41]Receptive Fields, Binocular Interaction and Functional Architecture in the Cat's Visual Cortex

D. H. Hubel, T. N. Wiesel - 1962

8 papers in library cite

[42]Unsupervised Learning of Distributions on Binary Vectors Using Two Layer Networks

Y. Freund, D. Haussler - 1992

8 papers in library cite

[43]A Global Geometric Framework for Nonlinear Dimensionality Reduction

J. Tenenbaum, V. D. Silva, John Langford - 2000

7 papers in library cite

[44]On the Power of Small-Depth Threshold Circuits

J. Hastad, M. Goldmann - 1991

7 papers in library cite

[45]The Curse of Highly Variable Functions for Local Kernel Machines

Yoshua Bengio, O. Delalleau, N. L. Roux - 2006

7 papers in library cite

[46]A Learning algorithm for Boltzmann Machines

D. H. Ackley, Geoffrey E. Hinton, T. J. Sejnowski - 1985

6 papers in library cite

[47]Justifying and Generalizing Contrastive Divergence

Yoshua Bengio, O. Delalleau - 2007

5 papers in library cite

[48]Many-Layered Learning

P. Utgoff, D. Stracuzzi - 2002

5 papers in library cite

[49]Nonlinear Dimensionality Reduction by Locally Linear Embedding

S. T. Roweis, L. K. Saul - 2000

5 papers in library cite

[50]Random Forests

L. Breiman - 2001

5 papers in library cite

[51]Semantic Hashing

Ruslan Salakhutdinov, Geoffrey Hinton - 2007

5 papers in library cite

[52]Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions

X. Zhu, Zoubin Ghahramani, J. Lafferty - 2003

5 papers in library cite

[53]Almost Optimal Lower Bounds for Small Depth Circuits

J. Hastad - 1986

4 papers in library cite

[54]Auto-Association by Multilayer Perceptrons and Singular Value Decomposition

H. Bourlard, Y. Kamp - 1988

4 papers in library cite

[55]Best Practices for Convolutional Neural Networks

Patrice Y. Simard, D. Steinkraus, John C. Platt - 2003

4 papers in library cite

[56]Energy-Based Models for Sparse Overcomplete Representations

Yee Whye Teh, M. Welling, S. Osindero, Geoffrey E. Hinton - 2003

4 papers in library cite

[57]Learning Long-Term Dependencies Is Not as Difficult With NARX Recurrent Neural Networks

T. Lin, B. G. Horne, P. Tino, C. L. Giles - 1995

4 papers in library cite

[58]Natural Language Processing With Modular PDP Networks and Distributed Lexicon

R. Miikkulainen, M. G. Dyer - 1991

4 papers in library cite

[59]Stochastic Complexity in Statistical Inquiry

J. Rissanen - 1989

4 papers in library cite

[60]Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

Ruslan Salakhutdinov, Geoffrey E. Hinton - 2008

4 papers in library cite

[61]A Day of Great Illumination: B. F. Skinner's Discovery of Shaping

G. B. Peterson - 2004

3 papers in library cite

[62]A Logical Calculus of the Ideas Immanent in Nervous Activity

W. Mcculloch, W. Pitts - 1943

3 papers in library cite

[63]Boltzmann Machines, Constraint Satisfaction Networks That Learn

Geoffrey Hinton, T. Sejnowski, D. Ackley - 1984

3 papers in library cite

[64]Building Continuous Space Language Models for Transcribing European Languages

Holger Schwenk, Jean Luc Gauvain - 2005

3 papers in library cite

[65]Charting a Manifold

M. Brand - 2003

3 papers in library cite

[66]Decision Trees Do Not Generalize to New Variations

Yoshua Bengio, O. Delalleau, C. Simard - 2007

3 papers in library cite

[67]Exploiting Generative Models in Discriminative Classifiers

T. Jaakkola, D. Haussler - 1999

3 papers in library cite

[68]Learning Internal Representations

J. Baxter - 1994

3 papers in library cite

[69]Learning Multilevel Distributed Representations for High-Dimensional Sequences

Ilya Sutskever, Geoffrey E. Hinton - 2007

3 papers in library cite

[70]Learning With Local and Global Consistency

Denny Zhou, O. Bousquet, T. N. Lal, Jason Weston, B. Scholkopf - 2004

3 papers in library cite

[71]Mean Field Theory for Sigmoid Belief Networks

L. Saul, T. Jaakkola, M. Jordan - 1996

3 papers in library cite

[72]Modeling Human Motion Using Binary Latent Variables

Graham W. Taylor, Geoffrey E. Hinton, S. T. Roweis - 2007

3 papers in library cite

[73]Modeling Image Patches With a Directed Hierarchy of markov Random Field

S. Osindero, Geoffrey E. Hinton - 2008

3 papers in library cite

[74]Nonlinear Component Analysis as a Kernel Eigenvalue Problem

B. Scholkopf, A. Smola, Klaus Robert Muller - 1998

3 papers in library cite

[75]On Contrastive Divergence Learning

M. A. C. Perpinan, Geoffrey E. Hinton - 2005

3 papers in library cite

[76]Stacked Generalization

D. Wolpert - 1992

3 papers in library cite

[77]A Quantitative Theory of Immediate Visual Recognition

T. Serre, G. Kreiman, M. Kouh, C. Cadieu, U. Knoblich, T. Poggio - 2007

2 papers in library cite

[78]Active Learning With Statistical Models

D. Cohn, Zoubin Ghahramani, M. Jordan - 1995

2 papers in library cite

[79]Advances in Kernel Methods Support Vector Learning

B. Scholkopf, C. J. C. Burges, A. J. Smola - 1999

2 papers in library cite

[80]An Information Maximisation Approach to Blind Separation and Blind Deconvolution

A. Bell, T. Sejnowski - 1995

2 papers in library cite

[81]An Introduction to Hidden Markov Models

L. R. Rabiner, B. Juang - 1986

2 papers in library cite

[82]Analysis of a Complex of Statistical Variables Into Principal Components

H. Hotelling - 1933

2 papers in library cite

[83]Classification and Regression Trees

L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone - 1984

2 papers in library cite

[84]Constituent Parsing With Incremental Sigmoid Belief Networks

I. Titov, J. Henderson - 2007

2 papers in library cite

[85]Efficient Non-Parametric Function Induction in Semi-Supervised Learning

O. Delalleau, Yoshua Bengio, N. L. Roux - 2005

2 papers in library cite

[86]Gaussian Processes for Regression

C. Williams, C. Rasmussen - 1996

2 papers in library cite

[87]Global Continuation for Distance Geometry Problems

Ziyi Wu - 1997

2 papers in library cite

[88]Learning Eigenfunctions Links Spectral Embedding and Kernel PCA

Yoshua Bengio, O. Delalleau, N. L. Roux, J. F. Paiement, Pascal Vincent, M. Ouimet - 2004

2 papers in library cite

[89]Nonlinear Autoassociation Is Not Equivalent to PCA

N. Japkowicz, S. J. Hanson, M. A. Gluck - 2000

2 papers in library cite

[90]Numerical Continuation Methods. An Introduction

E. L. Allgower, K. Georg - 1980

2 papers in library cite

[91]Parallel Continuation-Based Global Optimization for Molecular Conformation and Protein Folding

T. Coleman, Ziyi Wu - 1994

2 papers in library cite

[92]Regularization and Semi-Supervised Learning on Large Graphs

M. Belkin, I. Matveeva, P. Niyogi - 2004

2 papers in library cite

[93]Reinforcement Today

B. F. Skinner - 1958

2 papers in library cite

[94]Segmentation Using Eigenvectors: A Unifying View

Y. Weiss - 1999

2 papers in library cite

[95]Separating the Polynomial-Time Hierarchy by Oracles

A. Yao - 1985

2 papers in library cite

[96]Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images

S. Geman, D. Geman - 1984

2 papers in library cite

[97]The Complexity of Boolean Functions

I. Wegener - 1987

2 papers in library cite

[98]The Helmholtz Machine

Peter Dayan, Geoffrey Hinton, R. Neal, R. Zemel - 1995

2 papers in library cite

[99]Unsupervised Learning of Image Transformations

R. Memisevic, Geoffrey Hinton - 2007

2 papers in library cite

[100]Using Manifold Structure for Partially Labeled Classification

M. Belkin, P. Niyogi - 2003

2 papers in library cite

[101]A Bayesian/Information Theoretic Model of Learning via Multiple Task Sampling

J. Baxter - 1997

1 paper in library cites

[102]A Context-Sensitive Generalization of ICA

B. Pearlmutter, L. Parra - 1996

1 paper in library cites

[103]A Formal Theory of Inductive Inference

R. J. Solomonoff - 1964

1 paper in library cites

[104]A New View of ica

Geoffrey Hinton, M. Welling, Y. Teh, S. Osindero - 2001

1 paper in library cites

[105]A Sparse and Locally Shift Invariant Feature Extractor Applied to Document Images

Marc'aurelio Ranzato, Yann Lecun - 2007

1 paper in library cites

[106]A Survey of Kernels for Structured Data

T. Gartner - 2003

1 paper in library cites

[107]A Tutorial on Energy-Based Learning

Yann Lecun, S. Chopra, Raia Hadsell, M. A. Ranzato, Fu Jie Huang - 2006

1 paper in library cites

[108]A Unified Energy-Based Framework for Unsupervised Learning

Marc'aurelio Ranzato, Y. Boureau, S. Chopra, Yann Lecun - 2007

1 paper in library cites

[109]An Information Measure for Classification

C. Wallace, D. Boulton - 1968

1 paper in library cites

[110]An Introduction to Kolmogorov Complexity and Its Applications. Second Edition

M. Li, P. Vitanyi - 1997

1 paper in library cites

[111]An Introduction to McMc for Machine Learning

C. Andrieu, N. D. Freitas, A. Doucet, M. Jordan - 2003

1 paper in library cites

[112]Complexity Lower Bounds for Approximation Algebraic Computation Trees

F. Cucker, D. Grigoriev - 1999

1 paper in library cites

[113]Computational Complexity of Neural Networks: A Survey

P. Orponen - 1994

1 paper in library cites

[114]Experiments With a New Boosting algorithm

Y. Freund, R. E. Schapire - 1996

1 paper in library cites

[115]Global Data Analysis and the Fragmentation Problem in Decision Tree Induction

R. Vilalta, G. Blix, L. Rendell - 1997

1 paper in library cites

[116]How to Make a Low-Dimensional Representation Suitable for Diverse Tasks

N. Intrator, S. Edelman - 1996

1 paper in library cites

[117]Input Space Versus Feature Space

B. Scholkopf, S. Mika, C. Burges, P. Knirsch, Klaus Robert Muller, G. Ratsch, A. Smola - 1999

1 paper in library cites

[118]Introduction to Mathematical Logic, 4th Ed

E. Mendelson - 1997

1 paper in library cites

[119]Learning Despite Concept Variation by Finding Structure in Attribute-Based Data

Ethan Perez, L. A. Rendell - 1996

1 paper in library cites

[120]Learning Kernel Parameters by Using Class Separability Measure

Lisa Wang, K. L. Chan - 2002

1 paper in library cites

[121]Learning Nonlinear Overcomplete Representations for Efficient Coding

M. Lewicki, T. Sejnowski - 1998

1 paper in library cites

[122]Learning the Kernel Matrix With Semi-Definite Programming

G. Lanckriet, N. Cristianini, P. Bartlett, L. E. Gahoui, M. Jordan - 2002

1 paper in library cites

[123]Links Between Perceptrons, MLPS and SVMS

Ronan Collobert, Samy Bengio - 2004

1 paper in library cites

[124]Loss Functions for Discriminative Training of Energy-Based Models

Yann Lecun, F. Huang - 2005

1 paper in library cites

Reference title contains 'et al'

[125]On Kernel-Target Alignment

N. Cristianini, J. S. Taylor, A. Elisseeff, J. Kandola - 2002

1 paper in library cites

[126]Random Decision Forest

T. K. Ho - 1995

1 paper in library cites

[127]Rational Kernels: Theory and Algorithms

C. Cortes, Patrick Haffner, M. Mohri - 2004

1 paper in library cites

[128]Representational Power of Restricted boltzmann Machines and Deep Belief Networks

N. L. Roux, Yoshua Bengio - 2008

1 paper in library cites

[129]Self-Supervised Boosting

M. Welling, R. Zemel, Geoffrey E. Hinton - 2003

1 paper in library cites

[130]Smoothing Techniques for Macromolecular Global Optimization

J. More, Ziyi Wu - 1996

1 paper in library cites

[131]The Entire Regularization Path for the Support Vector Machine

T. Hastie, S. Rosset, R. Tibshirani, Jiacheng Zhu - 2004

1 paper in library cites

[132]Three Approaches to the Quantitative Definition of Information

Andrei N. Kolmogorov - 1965

1 paper in library cites

[133]Transformation Invariant Autoassociation With Application to Handwritten Character Recognition

Holger Schwenk, M. Milgram - 1995

1 paper in library cites

[134]Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability

M. Hutter - 2005

1 paper in library cites

Cited by

25

papers in your library

Cites

34

papers in your library

Read

on July 17, 2025

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

Tags

Paper Aliases

No aliases