Papperoni

2006

To Recognize Shapes, First Learn to Generate Images

Geoffrey Hinton

Open PDF Google Scholar

citations

Cite Score

23

AI summary

This paper proposes hybrid methods combining several learning algorithms to efficiently train multilayer neural networks for shape recognition, leveraging unsupervised pretraining and achieving a new state-of-the-art error rate on the permutation-invariant MNIST task.

Main Contributions

Identified five strategies for learning multilayer networks: denial, evolution, procrastination, calculus (backpropagation), and top-down generative connections (wake-sleep algorithm).
Proposed a hybrid approach combining backpropagation and wake-sleep with a pre-training phase using unsupervised layer-by-layer learning.
Demonstrated that pre-training significantly improves classification performance on test data, particularly for deep networks.
Achieved a new record of 0.97% test error rate on the permutation-invariant MNIST task by combining contrastive wake-sleep fine-tuning with backpropagation gradients.
Showed that greedy layer-by-layer learning in a generative model can raise a lower bound on the log probability of the training data.

Abstract

The uniformity of the cortical architecture and the ability of functions to move to different areas of cortex following early damage strongly suggests that there is a single basic learning algorithm for extracting underlying structure from richly-structured, high-dimensional sensory data. There have been many attempts to design such an algorithm, but until recently they all suffered from serious computational weaknesses. This chapter describes several of the proposed algorithms and shows how they can be combined to produce hybrid methods that work efficiently in networks with many layers and millions of adaptive connections.

Citation Graph

Loading graph...

References [25]

Sort:

Filter:

[1]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[2]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[3]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[4]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[5]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[6]Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis

John C. Platt - 2003

12 papers in library cite

Very good paper. Simple and pragmatic.

[7]Learning Methods for Generic Object Recognition With Invariance to Pose and Lighting

Yann Lecun, Fu Jie Huang, Leon Bottou - 2004

18 papers in library cite

Good paper, nice methodology for creating different images. However, I think that this was not too impactful... I don't see this being used a lot.

[8]Efficient Learning of Sparse Representations With an Energy-Based Model

Marc'aurelio Ranzato, C. Poultney, S. Chopra, Yann Lecun - 2006

20 papers in library cite

It's ok. Not really good, but alright.

[9]The Wake-Sleep Algorithm for Unsupervised Neural Networks

Geoffrey Hinton, Peter Dayan, B. Frey, R. Neal - 1995

9 papers in library cite

It's okay... I get the feeling that this is early autoencoders work, but the term still didn't exist. I don't think it adds nothing new though.

[10]Une procédure d'apprentissage pour Réseau a seuil Asymmetrique (A Learning Scheme for Asymmetric Threshold Networks)

Yann Lecun - 1985

4 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[11]Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences

P. Werbos - 1974

14 papers in library cite

Werbos PhD Thesis, introduced backprop

[12]The Nature of Statistical Learning Theory

V. Vapnik - 1995

9 papers in library cite

Book, 300+ pages

[13]Learning-Logic

D. B. Parker - 1985

8 papers in library cite

[14]Principles of Neurodynamics

F. Rosenblatt - 1962

7 papers in library cite

[15]Training Invariant Support Vector Machines

D. Decoste, B. Scholkopf - 2002

6 papers in library cite

[16]Applied Optimal Control

J. A. E. Bryson, Y. C. Ho - 1969

4 papers in library cite

[17]Perceptron: An Introduction to Computational Geometry

M. Minsky, S. Papert - 1969

4 papers in library cite

[18]Pandemonium: A Paradigm for Learning

Oliver G. Selfridge - 1958

2 papers in library cite

[19]A More Biologically Plausible Learning Rule for Neural Networks

P. Mazzoni, R. A. Andersen, M. Jordan - 1991

1 paper in library cites

[20]Dependence on REM Sleep of Overnight Improvement of a Perceptual Skill

A. Karni, D. Tanne, B. Rubenstein, J. Askenasy, D. Sagi - 1994

1 paper in library cites

[21]Induction of Visual Orientation Modules in Auditory Cortex

J. Sharma, A. Angelucci, M. Sur - 2000

1 paper in library cites

[22]Learning in Spiking Neural Networks by Reinforcement of Stochastic Synaptic Transmission

H. S. Seung - 2003

1 paper in library cites

[23]Learning the Relationship Between Spoken Digits and Digit Images

A. Kaganov, S. Osindero, Geoffrey E. Hinton - 2006

1 paper in library cites

[24]Topographic Reorganization of Somatosensory Cortical Areas 3b and 1 in Adult Monkeys Following Restricted Deafferentation

M. Merzenich, J. Kaas, J. Wall, R. Nelson, M. Sur, D. Felleman - 1983

1 paper in library cites

[25]Weight Perturbation: An Optimal Architecture and Learning Techniquefor Analog VLSI Feedforward and Recurrent Multilayer Networks

M. Jabri, B. Flower - 1992

1 paper in library cites

Cited by

5

papers in your library

Cites

11

papers in your library

Read

on February 7, 2026

Maybe the best explanation of deep belief nets and RBMs by Hinton.

Tags

Paper Aliases

No aliases