Papperoni

1998

Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner

Open PDF Google Scholar

citations

Cite Score

99

AI summary

This paper reviews gradient-based learning methods for handwritten character recognition, highlighting convolutional neural networks for 2D shape variability. It introduces graph transformer networks (GTNs) for training multi-module systems globally and describes systems for online handwriting and bank check recognition, achieving record accuracy.

Main Contributions

Review of gradient-based learning techniques for handwritten character recognition.
Demonstrates the effectiveness of convolutional neural networks in handling the variability of 2D shapes.
Introduction of graph transformer networks (GTNs) as a learning paradigm for training multi-module systems globally.
Description of two systems for online handwriting recognition, showcasing the advantages of global training and the flexibility of GTNs.
Application of GTNs to reading bank checks, achieving record accuracy through the combination of convolutional neural networks and global training techniques.

Abstract

Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient-based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of two dimensional (2-D) shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation, recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN's), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank check is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal checks. It is deployed commercially and reads several million checks per day.

Citation Graph

Loading graph...

References [65]

Sort:

Filter:

[1]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[2]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[3]Handwritten Digit Recognition With a Back-Propagation Network

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel - 1990

10 papers in library cite

Just a minor derivation of the previous paper, adding more layers to the net. Pretty much the same exact text than the other one. Uninteresting if you read the other.

[4]Phoneme Recognition Using Time-Delay Neural Networks

A. H. Waibel, T. Hanazawa, Geoffrey Hinton, K. Shikano, K. Lang - 1989

13 papers in library cite

I didn't really like this paper, although it's a good early example of using backprop. I think it can be thought as a 1D convolution.

[5]Generalization and Network Design Strategies

Yann Lecun - 1989

5 papers in library cite

It's a nice intro to LeCun's early work and how he got to the Conv Net

[6]Cognitron: A Self-Organizing Multilayered Neural Network

Kunihiko Fukushima - 1975

4 papers in library cite

Boring overall, and not as impactful as the Neocognitron.

[7]Comparison of Learning Algorithms for Handwritten Digit Recognition

V. Vapnik - 1995

2 papers in library cite

Good paper that gives an overview of what was SotA on digit recognition in the mid-90s. It's not so well written but it's easy to understand. I think that this is even better than LeCun's other paper to understand what's going on

[8]Improving the Convergence of Back-Propagation Learning With Second-Order Methods

S. Becker, Yann Lecun - 1988

9 papers in library cite

Surprisingly good. I thought this would be very math-heavy but in the end it's very easy to understand. Sadly results are underwhelming

[9]Multi-Digit Recognition Using a Space Displacement Neural Network

O. Matan, C. J. C. Burges, Yann Lecun, John S. Denker - 1992

3 papers in library cite

Meh. They just use a CNN to read more digits, but doesn't seem to have a ton of impact.

[10]Learning in Random Nets

M. Minsky, Oliver G. Selfridge - 1961

1 paper in library cites

Minsky is a hater.

[11]A Learning Scheme for Asymmetric Threshold Networks

Yann Lecun - 1985

5 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[12]Learning Processes in an Asymmetric Threshold Network

Yann Lecun - 1986

3 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[13]Learning and Relearning in Boltzmann Machines

Geoffrey E. Hinton, T. J. Sejnowski - 1986

9 papers in library cite

37 pages; Introduced Boltzmann machines

[14]Statistical Learning Theory

V. N. Vapnik - 1998

10 papers in library cite

Book, but there's a 12 page overview

[15]Pattern Classification and Scene Analysis

R. O. Duda, P. E. Hart - 1973

9 papers in library cite

[16]The Nature of Statistical Learning Theory

V. Vapnik - 1995

9 papers in library cite

Book, 300+ pages

[17]Learning-Logic

D. B. Parker - 1985

8 papers in library cite

[18]Receptive Fields, Binocular Interaction and Functional Architecture in the Cat's Visual Cortex

D. H. Hubel, T. N. Wiesel - 1962

8 papers in library cite

[19]Neocognitron: A New algorithm for Pattern Recognition Tolerant of Deformations and Shifts in Position

Kunihiko Fukushima, S. Miyake - 1982

7 papers in library cite

[20]A Learning algorithm for Boltzmann Machines

D. H. Ackley, Geoffrey E. Hinton, T. J. Sejnowski - 1985

6 papers in library cite

[21]Applied Optimal Control

J. A. E. Bryson, Y. C. Ho - 1969

4 papers in library cite

[22]Maximum Mutual Information Estimation of Hidden Markov Model Parameters for Speech Recognition

L. Bahl, P. Brown, P. D. Souza, R. Mercer - 1986

4 papers in library cite

[23]Efficient Pattern Recognition Using a New Transformation Distance

Patrice Simard, Yann Lecun, John Denker - 1993

3 papers in library cite

[24]First- And Second-Order Methods for Learning: Between Steepest Descent and Newton's method

R. Battiti - 1992

3 papers in library cite

[25]A Framework for the Cooperation of Learning Algorithms

Leon Bottou, P. Gallinari - 1991

2 papers in library cite

[26]A Theoretical Framework for Back-Propagation

Yann Lecun - 1988

2 papers in library cite

[27]A Theory of Adaptive Pattern Classifiers

S. I. Amari - 1967

2 papers in library cite

[28]A Time Delay Neural Network Architecture for Speech Recognition

Kevin J. Lang, Geoffrey E. Hinton - 1988

2 papers in library cite

[29]Adaptation and Learning in Automatic Systems

Y. Tsypkin - 1971

2 papers in library cite

[30]Combining Hidden Markov Models and Neural Network Classifiers

L. T. Niles, H. F. Silverman - 1990

2 papers in library cite

[31]Efficient Parallel Learning Algorithms for Neural Networks

A. H. Kramer, A. S. Vincentelli - 1988

2 papers in library cite

[32]Eigenvalues of Covariance Matrices: Application to Neural-Network Learning

Yann Lecun, I. Kanter, Sara Solla - 1991

2 papers in library cite

[33]Improving the Accuracy and Speed of Support Vector Machines

C. J. C. Burges, B. Schoelkopf - 1997

2 papers in library cite

[34]Integrated Segmentation and Recognition of Hand-Printed Numerals

J. Keeler, D. Rumelhart, W. K. Leow - 1991

2 papers in library cite

[35]Learning Curves: Asymptotic Values and Rate of Convergence

C. Cortes, Lawrence Jackel, Sara Solla, V. N. Vapnik, John Denker - 1993

2 papers in library cite

[36]Modèles Connexionnistes De l'Apprentissage (Connectionist Learning Models)

Yann Lecun - 1987

2 papers in library cite

[37]Numerical Recipes: The Art of Scientific Computing

W. H. Press, B. P. Flannery, S. A. Teukolsky, W. T. Vetterling - 1986

2 papers in library cite

[38]Signature Verification Using a Siamese Time Delay Neural Network

J. Bromley, J. W. Bentz, Leon Bottou, I. Guyon, Yann Lecun, C. Moore, E. Sackinger, R. Shah - 1993

2 papers in library cite

[39]Solving Multiclass Learning Problems via Error-Correcting Output Codes

T. G. Dietterich, G. Bakiri - 1995

2 papers in library cite

[40]A Connectionist Recognizer for on-Line Cursive Handwriting Recognition

S. Manke, U. Bodenhausen - 1994

1 paper in library cites

[41]An EM algorithm for Asynchronous Input/Output Hidden Markov Models

Samy Bengio, Yoshua Bengio - 1996

1 paper in library cites

[42]Automatic Processing of Information on Checks

Lucas Lam, C. Y. Suen, D. Guillevic, N. W. Strathy, M. Cheriet, K. Liu, J. N. Said - 1995

1 paper in library cites

[43]Centered-Object Integrated Segmentation and Recognition of Overlapping Hand-Printed Characters

G. L. Martin - 1993

1 paper in library cites

[44]Computer Recognition of Unconstrained Handwritten Numerals

C. Y. Suen, C. Nadal, R. Legault, T. A. Mai, Lucas Lam - 1992

1 paper in library cites

[45]Cursive Script Recognition Applied to the Processing of Bank Checks

D. Guillevic, C. Y. Suen - 1995

1 paper in library cites

[46]Design of a Neural Network Character Recognizer for a Touch Terminal

I. Guyon, P. Albrecht, Yann Lecun, John S. Denker, W. Hubbard - 1991

1 paper in library cites

[47]Discriminative Learning for Minimum Error Classification

B. H. Juang, S. Katagiri - 1992

1 paper in library cites

[48]Efficient Training of Feed-Forward Neural Networks

M. Moller - 1993

1 paper in library cites

[49]Fast Neural Net Simulation With a DSP Processor Array

U. Muller, A. Gunzinger, W. Guggenbuhl - 1995

1 paper in library cites

[50]Foundations of the Theory of Learning Systems

Y. Tsypkin - 1973

1 paper in library cites

[51]Handwritten Digit Recognition: Applications of Neural Net Chips and Automatic Learning

Yann Lecun, L. D. Jackel, B. Boser, John S. Denker, H. P. Graf, I. Guyon, D. Henderson, R. E. Howard, W. Hubbard - 1989

1 paper in library cites

[52]High-Performance Reading Machines

S. N. Srihari - 1992

1 paper in library cites

[53]Measuring the Vc-Dimension of a Learning Machine

V. N. Vapnik, E. Levin, Yann Lecun - 1994

1 paper in library cites

[54]Multi-Resolution Neural Networks for Omnifont Character Recognition

J. Wang, J. Jean - 1993

1 paper in library cites

[55]Multi-State Time-Delay Neural Networks for Continuous Speech Recognition

Patrick Haffner, A. H. Waibel - 1992

1 paper in library cites

[56]Neural Networks for Speech and Sequence Recognition

Yoshua Bengio - 1996

1 paper in library cites

[57]Off Line Recognition of Handwritten Postal Words Using Neural Networks

C. J. C. Burges, J. I. Ben, John S. Denker, Yann Lecun, C. R. Nohl - 1993

1 paper in library cites

[58]On-Line Handwriting Recognition With Neural Networks: Spatial Representation Versus Temporal Representation

Yann Lecun, Yoshua Bengio, D. Henderson, A. Weisbuch, H. Weissman, Lawrence Jackel - 1993

1 paper in library cites

[59]Recognition of Cursive Script Amounts on Postal Checks

M. Gilloux, M. Leroux - 1993

1 paper in library cites

[60]Speaker Independent Isolated Digit Recognition: Multilayer perceptron Versus Dynamic Time Warping

Leon Bottou, F. Fogelman, P. Blanchet, J. S. Lienard - 1990

1 paper in library cites

[61]Speech Recognition With Continuous-Parameter Hidden Markov Models

L. R. Bahl, P. F. Brown, P. V. D. Souza, R. L. Mercer - 1987

1 paper in library cites

[62]Statistical Mechanics of Learning From Examples

S. Seung, H. Sompolinsky, N. Tishby - 1992

1 paper in library cites

[63]The Perception of Multiple Objects: A Connectionist Approach

M. C. Mozer - 1991

1 paper in library cites

[64]The State of the Art in on-Line Handwriting Recognition

C. Tappert, C. Suen, T. Wakahara - 1990

1 paper in library cites

[65]Time-Delay Neural Networks Embedding Time Alignment: A Performance Analysis

Patrick Haffner, A. H. Waibel - 1991

1 paper in library cites

Cited by

62

papers in your library

Cites

14

papers in your library

Read

on June 24, 2025

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

Tags

Paper Aliases

The MNIST Database of Handwritten Digits