Papperoni

1987

Connectionist Learning Procedures

Geoffrey E. Hinton

Open PDF Google Scholar

citations

Cite Score

62

AI summary

This paper surveys various connectionist learning procedures including supervised, unsupervised, and reinforcement methods, highlighting their mechanisms, applications in tasks like semantic feature discovery and speech recognition, and discussions on generalization and learning speed, particularly focusing on gradient-descent techniques for complex internal representations.

Main Contributions

Surveyed efficient learning procedures for connectionist networks to construct complex internal representations by modifying connection strengths.
Described gradient-descent procedures where each connection computes the derivative of a global error measure and adjusts strength to decrease error.
Discussed "backpropagation" as a multi-layer least squares procedure, demonstrating its ability to discover semantic features and its application in text-to-speech mapping and phoneme recognition.
Introduced Boltzmann Machines as a generalization of Hopfield nets for learning complex tasks using hidden units, and methods for maximizing reinforcement and entropy.
Explored self-supervised backpropagation, competitive learning, and reinforcement learning procedures, including their deficiencies and practical methods for improving generalization and speed.

Abstract

A major goal of research on networks of neuron-like processing units is to discover efficient learning procedures that allow these networks to construct complex internal representations of their environment. The learning procedures must be capable of modifying the connection strengths in such a way that internal units which are not part of the input or output come to represent important features of the task domain. Several interesting gradient-descent procedures have recently been discovered. Each connection computes the derivative, with respect to the connection strength, of a global measure of the error in the performance of the network. The strength is then adjusted in the direction that decreases the error. These relatively simple, gradient-descent learning procedures work well for small tasks and the new challenge is to find ways of improving their convergence rate and their generalization abilities so that they can be applied to larger, more realistic tasks.

Citation Graph

Loading graph...

References [104]

Sort:

Filter:

[1]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[2]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[3]Neural Networks and Physical Systems With Emergent Collective Computational Abilities

J. J. Hopfield - 1982

8 papers in library cite

Very theory heavy and doesn't seem to add much... Famous because it introduced the "Hopfield network" - "At a time when research in neural networks was facing a lull, Hopfield's paper injected a fresh perspective by demonstrating how a network of simple, interconnected units could exhibit complex computational abilities as a collective, emergent property."

[4]Phoneme Recognition Using Time-Delay Neural Networks

A. H. Waibel, T. Hanazawa, Geoffrey Hinton, K. Shikano, K. Lang - 1989

13 papers in library cite

I didn't really like this paper, although it's a good early example of using backprop. I think it can be thought as a 1D convolution.

[5]Distributed Representations

Geoffrey E. Hinton, J. L. Mcclelland, D. E. Rumelhart - 1986

5 papers in library cite

Very relevant but also very abstract. Not really an awesome read, but interesting nonetheless.

[6]Learning Distributed Representations of Concepts

Geoffrey E. Hinton - 1986

13 papers in library cite

 Probably seminal, but a bit boring overall

[7]Cognitron: A Self-Organizing Multilayered Neural Network

Kunihiko Fukushima - 1975

4 papers in library cite

Boring overall, and not as impactful as the Neocognitron.

[8]A Learning Scheme for Asymmetric Threshold Networks

Yann Lecun - 1985

5 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

M. Minsky, S. Papert - 1969

12 papers in library cite

Book, 500 pages

[10]Learning and Relearning in Boltzmann Machines

Geoffrey E. Hinton, T. J. Sejnowski - 1986

9 papers in library cite

37 pages; Introduced Boltzmann machines

[11]Parallel Models of Associative Memory

Geoffrey E. Hinton, J. A. Anderson - 1981

4 papers in library cite

This is actually a book, I only found chapter 6 online

[12]Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences

P. Werbos - 1974

14 papers in library cite

Werbos PhD Thesis, introduced backprop

[13]Parallel Distributed Processing: Explorations in the Microstructure of Cognition, I: Foundations

D. E. Rumelhart, J. L. Mcclelland - 1986

1 paper in library cites

[14]Modèles Connexionnistes De l'Apprentissage

Yann Lecun - 1987

9 papers in library cite

[15]Learning-Logic

D. B. Parker - 1985

8 papers in library cite

[16]Neocognitron: A New algorithm for Pattern Recognition Tolerant of Deformations and Shifts in Position

Kunihiko Fukushima, S. Miyake - 1982

7 papers in library cite

[17]Principles of Neurodynamics

F. Rosenblatt - 1962

7 papers in library cite

[18]A Learning algorithm for Boltzmann Machines

D. H. Ackley, Geoffrey E. Hinton, T. J. Sejnowski - 1985

6 papers in library cite

[19]Maximum Likelihood From Incomplete Data via the EM Algorithm

A. P. Dempster, N. M. Laird, D. B. Rubin - 1977

6 papers in library cite

[20]Optimization by Simulated Annealing

S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi - 1983

6 papers in library cite

[21]Adaptive Switching Circuits

B. Widrow, H. E. Hoff - 1960

5 papers in library cite

[22]The Organization of Behavior

D. Hebb - 1949

5 papers in library cite

[23]A Maximum Likelihood Approach to Continuous Speech Recognition

L. R. Bahl, Frederick Jelinek, R. L. Mercer - 1983

4 papers in library cite

[24]Adaptation in Natural and Artificial Systems

J. H. Holland - 1975

4 papers in library cite

[25]Adaptive Signal Processing

B. Widrow, S. D. Stearns - 1985

4 papers in library cite

[26]Associative Memory: A System-Theorectical Approach

T. Kohonen - 1977

3 papers in library cite

[27]Learning Representations by Recirculation

Geoffrey E. Hinton, J. L. Mcclelland - 1988

3 papers in library cite

[28]Learning the Hidden Structure of Speech

Jeffrey L. Elman, David Zipser - 1987

3 papers in library cite

[29]Learning to Predict by the Methods of Temporal Differences

Richard S. Sutton - 1988

3 papers in library cite

[30]Neuronlike Elements That Can Solve Difficult Learning Control Problems

A. G. Barto, Richard S. Sutton, C. W. Anderson - 1983

3 papers in library cite

[31]Parallel Networks That Learn to Pronounce English Text

T. J. Sejnowski, C. R. Rosenberg - 1987

3 papers in library cite

[32]Reinforcement Learning in Connectionist Networks: A Mathematical Analysis

Ronald J. Williams - 1986

3 papers in library cite

[33]Self-Organization of Orientation Sensitive Cells in the Striate Cortex

C. V. D. Malsburg - 1973

3 papers in library cite

[34]A Dual Back-Propagation Scheme for Scalar Reward Learning

P. Munro - 1987

2 papers in library cite

[35]A Theory of Adaptive Pattern Classifiers

S. I. Amari - 1967

2 papers in library cite

[36]A Theory of the Learnable

L. G. Valiant - 1984

2 papers in library cite

[37]A Unified Framework for Connectionist Systems

R. M. Golden - 1988

2 papers in library cite

[38]Boltzmann Machines for Speech Recognition

R. W. Prager, T. D. Harrison, F. Fallside - 1986

2 papers in library cite

[39]Holography, Associative Memory, and Inductive Generalization

D. Willshaw - 1981

2 papers in library cite

[40]How Does a Brain Build a Cognitive Code?

S. Grossberg - 1980

2 papers in library cite

[41]Implementing Semantic Networks in Parallel Hardware

Geoffrey Hinton - 1981

2 papers in library cite

[42]Learning Sets of Filters Using Back-Propagation

D. C. Plaut, Geoffrey E. Hinton - 1987

2 papers in library cite

[43]Neural Computation by Concentrating Information in Time

D. W. Tank, J. J. Hopfield - 1987

2 papers in library cite

[44]Pattern Recognizing Stochastic Learning automata

A. G. Barto, P. Anandan - 1985

2 papers in library cite

[45]Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology

H. B. Barlow - 1972

2 papers in library cite

[46]Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images

S. Geman, D. Geman - 1984

2 papers in library cite

[47]Temporal Credit Assignment in Reinforcement Learning

Richard S. Sutton - 1984

2 papers in library cite

[48]Using Fast Weights to deblur Old Memories

Geoffrey Hinton, D. Plaut - 1987

2 papers in library cite

[49]A Complexity Level Analysis of Vision

J. K. Tsotsos - 1987

1 paper in library cites

[50]A Marker Induction Mechanism for the Establishment of Ordered Neural Mapping: Its Application to the Retino-Tectal Connections

D. J. Willshaw, C. V. D. Malsburg - 1979

1 paper in library cites

[51]A Mean Field Theory Learning algorithm for Neural Networks

C. Peterson, J. R. Anderson - 1987

1 paper in library cites

[52]A Neuromorphic VLSI Learning System

J. Alspector, R. B. Allen - 1987

1 paper in library cites

[53]A Theory for the Acquisition and Loss of Neuron Specificity in Visual Cortex

L. N. Cooper, F. Liberman, E. Oja - 1979

1 paper in library cites

[54]Abstraction and Representation of Continuous Variables in Connectionist Networks

E. Saund - 1986

1 paper in library cites

[55]Adaptive Pattern Classification and Universal Recoding, I: Parallel Development and Coding of Neural Feature Detectors

S. Grossberg - 1976

1 paper in library cites

[56]Categorization and Selective Neurons

J. A. Anderson, M. C. Mozer - 1981

1 paper in library cites

[57]Clustering, Taxonomy, and Topological Maps of Patterns

T. Kohonen - 1982

1 paper in library cites

[58]Competitive Learning

D. E. Rumelhart, David Zipser - 1985

1 paper in library cites

[59]Complexity of Connectionist Learning With Various Node Functions

J. S. Judd - 1987

1 paper in library cites

[60]Connectionist Models and Their Properties

J. A. Feldman, D. H. Ballard - 1982

1 paper in library cites

[61]Connectionist Models of Boolean Category Representation

D. J. Volper, S. E. Hampson - 1986

1 paper in library cites

[62]Connectionist Speech Recognition

Kevin J. Lang - 1987

1 paper in library cites

[63]Cortical Connections and Parallel Processing: Structure and Function

D. H. Ballard - 1986

1 paper in library cites

[64]Development of Feature-Analyzing Cells and Their Columnar Organization in a Layered Self-Adaptive Network

R. Linsker - 1987

1 paper in library cites

[65]Disjunctive Models of Boolean Category Learning

S. E. Hampson, D. J. Volper - 1987

1 paper in library cites

[66]Dynamic Connections in Neural Networks

J. A. Feldman - 1982

1 paper in library cites

[67]Field Theory of Self-Organizing Neural Nets

S. I. Amari - 1983

1 paper in library cites

[68]From Basic Network Principles to Neural Architecture: Emergence of Orientation Columns

R. Linsker - 1986

1 paper in library cites

[69]From Basic Network Principles to Neural Architecture: Emergence of Orientation-Selective Cells

R. Linsker - 1986

1 paper in library cites

[70]From Basic Network Principles to Neural Architecture: Emergence of Spatial Opponent Cells

R. Linsker - 1986

1 paper in library cites

[71]Fusion, Propagation, and Structuring in Belief Networks

J. Pearl - 1986

1 paper in library cites

[72]G-Maximization: An Unsupervised Learning Procedure for Discovering Regularities

B. A. Pearlmutter, Geoffrey E. Hinton - 1986

1 paper in library cites

[73]Genetic Algorithms and Simulated Annealing

L. Davis - 1987

1 paper in library cites

[74]How Learning Can Guide Evolution

Geoffrey E. Hinton, S. J. Nowlan - 1987

1 paper in library cites

[75]Inductive Inference: Theory and Methods

D. Angluin, C. H. Smith - 1983

1 paper in library cites

[76]Learning Internal Representations From Gray-Scale Images: An Example of Extensional Programming

G. W. Cottrell, P. Munro, David Zipser - 1987

1 paper in library cites

[77]Learning Optimal Discriminant Functions Through a Cooperative Game of Automata

M. A. L. Thatachar, P. S. Sastry - 1985

1 paper in library cites

[78]Learning Translation Invariant Recognition in a Massively Parallel Network

Geoffrey E. Hinton - 1987

1 paper in library cites

[79]Models of Information Processing in the Brain

J. A. Anderson, Geoffrey E. Hinton - 1981

1 paper in library cites

[80]Modular Learning in Neural Networks

D. H. Ballard - 1987

1 paper in library cites

[81]Multiresolution Computation of Visible Surface Representations

D. Terzopoulos - 1984

1 paper in library cites

[82]Neural Computation of Decisions in Optimization Problems

J. J. Hopfield, D. W. Tank - 1985

1 paper in library cites

[83]Neural Representation of Conceptual Knowledge

J. A. Feldman - 1986

1 paper in library cites

[84]On the Acquisition of the Past Tense in English

D. E. Rumelhart, J. L. Mcclelland - 1986

1 paper in library cites

[85]On the Foundations of Relaxation Labeling Processes

R. A. Hummel, S. W. Zucker - 1983

1 paper in library cites

[86]Optimal Perceptual Inference

Geoffrey E. Hinton, T. J. Sejnowski - 1983

1 paper in library cites

[87]Optimization Strategies Gleaned From Biological Evolution

R. M. Brady - 1985

1 paper in library cites

[88]Parallel Distributed Processing: Explorations in the Microstructure of Cognition, II: Applications

D. E. Rumelhart, J. L. Mcclelland - 1986

1 paper in library cites

[89]Parallel Visual Computation

D. H. Ballard, Geoffrey E. Hinton, T. J. Sejnowski - 1983

1 paper in library cites

[90]Plain Talk About Neurodevelopmental Epistemology

M. L. Minsky - 1977

1 paper in library cites

[91]Probabilistic Solution of Inverse Problems

J. L. Marroquin - 1985

1 paper in library cites

[92]Scaling Relationships in Back-Propagation Learning: Dependence on Training Set Size

Gerald Tesauro - 1987

1 paper in library cites

Missing year

[93]Second Order Back-Propagation: An Optimal Adaptive Algorithm for Any Adaptive Network

D. B. Parker

1 paper in library cites

[94]Selective Networks Capable of Representative Transformations, Limited Generalizations, and Associative Memory

G. M. Edelman, G. N. Reeke - 1982

1 paper in library cites

[95]Stochastic Iterated genetic Hill-Climbing

D. H. Ackley - 1987

1 paper in library cites

[96]The Correlation Theory of Brain Function

C. V. D. Malsburg - 1981

1 paper in library cites

[97]The Elastic Net Method: An Analogue Approach to the Travelling Salesman Problem

R. Durbin, D. Willshaw - 1987

1 paper in library cites

[98]The Evolution of Data Processing Abilities in Competing Automata

M. Kerszberg, A. Bergman - 1986

1 paper in library cites

[99]The Function of Dream Sleep

F. Crick, G. Mitchison - 1983

1 paper in library cites

[100]The Sciences of the Artificial

H. A. Simon - 1969

1 paper in library cites

[101]Theory for the Development of neuron Selectivity: Orientation Specificity and Binocular Interaction in Visual Cortex

E. L. Bienenstock, L. N. Cooper, P. W. Munro - 1982

1 paper in library cites

[102]Theory of Neural-Analog Reinforcement Systems and Its Application to the Brain-Model Problem

M. L. Minsky - 1954

1 paper in library cites

[103]Understanding Biological Computation: Reliable Learning and Recognition

T. Hogg, B. A. Huberman - 1984

1 paper in library cites

[104]Unlearning Has a Stabilizing Effect in Collective Memories

J. J. Hopfield, D. I. Feinstein, R. G. Palmer - 1983

1 paper in library cites

Cited by

11

papers in your library

Cites

13

papers in your library

Read

on January 25, 2026

It's a very good overview of everything that was happening in 1987! A bit too long though, but a good start nonetheless.

Tags

Paper Aliases

No aliases