Papperoni

1998

Efficient Backprop

Yann Lecun, Leon Bottou, G. B. Orr, Klaus Robert Muller

Open PDF Google Scholar

citations

Cite Score

79

AI summary

This paper analyzes back-propagation learning, offering tricks and explanations. It finds classical second-order methods impractical for large neural networks, suggesting alternatives. It emphasizes convergence improvement through techniques like stochastic learning, input normalization, and appropriate sigmoid functions, enhancing neural network training efficiency.

Main Contributions

Analysis of back-propagation learning to explain common phenomena.
Introduction of tricks to avoid undesirable behaviors of backprop.
Explanation of why the tricks work.
Demonstration that most classical second-order methods are impractical for large neural networks.
Proposal of a few second-order methods that do not have these limitations.

Abstract

The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most "classical" second-order methods are impractical for large neural networks. A few methods are proposed that do not have these limitations.

Citation Graph

Loading graph...

References [44]

Sort:

Filter:

[1]Handwritten Digit Recognition With a Back-Propagation Network

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel - 1990

10 papers in library cite

Just a minor derivation of the previous paper, adding more layers to the net. Pretty much the same exact text than the other one. Uninteresting if you read the other.

[2]Optimal Brain Damage

Yann Lecun, John Denker, Sara Solla, Richard Howard, Lawrence Jackel - 1990

4 papers in library cite

It's a nice idea but I think it's a bit uninteresting - maybe it influenced other important work later?

[3]Phoneme Recognition Using Time-Delay Neural Networks

A. H. Waibel, T. Hanazawa, Geoffrey Hinton, K. Shikano, K. Lang - 1989

13 papers in library cite

I didn't really like this paper, although it's a good early example of using backprop. I think it can be thought as a 1D convolution.

[4]Increased Rates of Convergence Through Learning Rate Adaptation

Robert A. Jacobs - 1988

4 papers in library cite

Very early work on learning rate adaptation. I like that it describes the concepts very thoroughly (as they are all new).

[5]Generalization and Network Design Strategies

Yann Lecun - 1989

5 papers in library cite

It's a nice intro to LeCun's early work and how he got to the Conv Net

[6]Improving the Convergence of Back-Propagation Learning With Second-Order Methods

S. Becker, Yann Lecun - 1988

9 papers in library cite

Surprisingly good. I thought this would be very math-heavy but in the end it's very easy to understand. Sadly results are underwhelming

[7]Neural Networks for Pattern Recognition

C. M. Bishop - 1995

12 papers in library cite

Book, 38k citations

[8]Statistical Learning Theory

V. N. Vapnik - 1998

10 papers in library cite

Book, but there's a 12 page overview

[9]The Nature of Statistical Learning Theory

V. Vapnik - 1995

9 papers in library cite

Book, 300+ pages

[10]Natural Gradient Works Efficiently in Learning

S. I. Amari - 1998

6 papers in library cite

[11]Fast Exact Multiplication by the Hessian

B. Pearlmutter - 1994

4 papers in library cite

[12]Online Algorithms and Stochastic Approximations

Leon Bottou - 1998

4 papers in library cite

[13]First- And Second-Order Methods for Learning: Between Steepest Descent and Newton's method

R. Battiti - 1992

3 papers in library cite

[14]Dynamics and Algorithms for Stochastic Search

G. B. Orr - 1996

2 papers in library cite

[15]Efficient Parallel Learning Algorithms for Neural Networks

A. H. Kramer, A. S. Vincentelli - 1988

2 papers in library cite

[16]Fast Learning in Networks of Locally-Tuned Processing Units

J. Moody, C. Darken - 1989

2 papers in library cite

[17]Modèles Connexionnistes De l'Apprentissage (Connectionist Learning Models)

Yann Lecun - 1987

2 papers in library cite

[18]Stochastic Dynamics of Learning With Momentum in Neural Networks

W. Wiegerinck, A. Komoda, T. Heskes - 1994

2 papers in library cite

[19](In Japanese)

N. Murata - 1992

1 paper in library cites

[20]A Scaled Conjugate Gradient algorithm for Fast Supervised Learning

M. Moller - 1993

1 paper in library cites

[21]Adapting Bias by Gradient Descent an Incremental Version of Delta-Bar-Delta

Richard S. Sutton - 1992

1 paper in library cites

[22]Adaptive on-Line Learning in Changing Environments

N. Murata, Klaus Robert Muller, A. Ziehe, S. Amari - 1997

1 paper in library cites

[23]Automatic Learning Rate Maximization by on-Line Estimation of the Hessian's Eigenvectors

Yann Lecun, Patrice Y. Simard, B. Pearlmutter - 1993

1 paper in library cites

[24]Computing Second Order Derivatives in Feed-Forward Networks a Review

W. L. Buntine, A. S. Weigend - 1993

1 paper in library cites

[25]Digital Signal Processing

A. V. Oppenheim, R. W. Schafer - 1975

1 paper in library cites

[26]Exact Solution for on-Line Learning in Multilayer Neural networks

D. Saad, Sara A. Solla - 1995

1 paper in library cites

[27]Matrix Computations

G. H. Golub, C. F. V. Loan - 1989

1 paper in library cites

[28]Mean Square Optimality in the Continuous Time Robbins Monro Procedure

L. Goldstein - 1987

1 paper in library cites

[29]Minimisation Methods for Training Feed-Forward Networks

P. V. D. Smagt - 1994

1 paper in library cites

[30]Multivariable Function Interpolation and Adaptive Networks

D. S. Broomhead, D. Lowe - 1988

1 paper in library cites

[31]Neural Learning in Structured Parameter Spaces Natural Riemannian Gradient

S. Amari - 1997

1 paper in library cites

[32]Neural Networks and the Bias/Variance Dilemma

S. Geman, E. Bienenstock, R. Doursat - 1992

1 paper in library cites

[33]Note on Learning Rate Schedules for Stochastic Optimization

C. Darken, J. E. Moody - 1991

1 paper in library cites

[34]Numerical Recipies in C: The Art of Scientific Programming

W. H. Press, B. P. Flannery, S. A. Teukolsky, W. T. Vetterling - 1988

1 paper in library cites

[35]On-Line Learning of Dichotomies: Algorithms and Learning Curves

H. Sompolinsky, N. Barkai, H. S. Seung - 1995

1 paper in library cites

[36]On-Line Learning Processes in Artificial Neural Networks

T. M. Heskes, B. Kappen - 1993

1 paper in library cites

[37]Online Learning in Neural Networks (1997 Workshop at the Newton Institute)

D. Saad - 1998

1 paper in library cites

[38]Practical Methods of Optimization

R. Fletcher - 1987

1 paper in library cites

[39]Principal Component Neural Networks

K. I. Diamantaras, S. Y. Kung - 1996

1 paper in library cites

[40]Regularization in the Selection of Radial Basis Function Centers

M. J. L. Orr - 1995

1 paper in library cites

[41]Removing Noise in on-Line Search Using Adaptive Batch Sizes

G. B. Orr - 1997

1 paper in library cites

[42]Second Order Properties of Error Surfaces

Yann Lecun, I. Kanter, Sara A. Solla - 1991

1 paper in library cites

[43]Supervised Learning on Large Redundant Training Sets

M. Moller - 1993

1 paper in library cites

[44]The Efficiency and the Robustness of Natural Gradient Descent Learning Rule

H. H. Yang, S. Amari - 1998

1 paper in library cites

Cited by

20

papers in your library

Cites

8

papers in your library

Read

on June 26, 2025

The first half is very very good. The remainder is very boring.

Tags

Paper Aliases

No aliases