Papperoni

2010

Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition

Dan C. Ciresan, Ueli Meier, Luca M. Gambardella, Jürgen Schmidhuber

Open PDF Google Scholar

citations

Cite Score

1

AI summary

This paper introduces a deep multi-layer perceptron (MLP) model trained with back-propagation on MNIST dataset using GPUs, achieving a 0.35% error rate through extensive training image deformations and graphics card acceleration.

Main Contributions

Demonstrated that deep, plain MLPs can achieve state-of-the-art results on MNIST with sufficient training and computational resources.
Achieved a low 0.35% error rate on the MNIST handwritten digits benchmark using online back-propagation.
Utilized graphics cards (GPUs) to significantly speed up the training process.
Employed extensive training image deformations to improve the network's generalization.
Showed that hardware progress is important in deep learning.

Abstract

Good old on-line back-propagation for plain multi-layer perceptrons yields a very low 0.35% error rate on the famous MNIST handwritten digits benchmark. All we need to achieve this best result so far are many hidden layers, many neurons per layer, numerous deformed training images, and graphics cards to greatly speed up learning.

Citation Graph

Loading graph...

References [21]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[3]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[4]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[5]Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis

John C. Platt - 2003

12 papers in library cite

Very good paper. Simple and pragmatic.

[6]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[7]Efficient Learning of Sparse Representations With an Energy-Based Model

Marc'aurelio Ranzato, C. Poultney, S. Chopra, Yann Lecun - 2006

20 papers in library cite

It's ok. Not really good, but alright.

[8]Unsupervised Learning of Invariant Feature Hierarchies With Applications to Object Recognition

Marc'aurelio Ranzato, F. Huang, Y. Boureau, Yann Lecun - 2007

8 papers in library cite

I thought that I had learned pretty much everything before 2015, but this paper proved me wrong. Amazing read and very nice methodology. Very simple as well!

[9]High Performance Convolutional Neural Networks for Document Processing

K. Chellapilla, S. Puri, Patrice Y. Simard - 2006

3 papers in library cite

It's ok. It's not bad, it's just a bit boring, and I don't think it's bringing anything new. People already implemented unrolled CNN and GPUs, they just brought it all together.

[10]Une Procédure D'Apprentissage pour Réseau à Seuil Asymétrique

Yann Lecun - 1985

5 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[11]To Recognize Shapes, First Learn to Generate Images

Geoffrey Hinton - 2006

5 papers in library cite

Maybe the best explanation of deep belief nets and RBMs by Hinton.

[12]Using GPUs for Machine Learning Algorithms

D. Steinkraus, I. Buck, Patrice Simard - 2005

3 papers in library cite

It's not bad, it's just not worth the read.

[13]Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences

P. Werbos - 1974

14 papers in library cite

Werbos PhD Thesis, introduced backprop

[14]Untersuchungen zu dynamischen neuronalen netzen

Sepp Hochreiter - 1991

18 papers in library cite

[15]Training Invariant Support Vector Machines

D. Decoste, B. Scholkopf - 2002

6 papers in library cite

[16]Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure

Ruslan Salakhutdinov, Geoffrey Hinton - 2007

5 papers in library cite

[17]Artificial Intelligence: A Modern Approach

S. Russell, P. Norvig - 1995

4 papers in library cite

[18]A Trainable Feature Extractor for Handwritten Digit Recognition

F. Lauer, C. Suen, G. Bloch - 2007

1 paper in library cites

[19]Accelerating Large-Scale Convolutional Neural Networks With Parallel Graphics Multiprocessors

D. Scherer, Sven Behnke - 2009

1 paper in library cites

[20]NVIDIA Cuda

Nvidia - 2009

1 paper in library cites

[21]Optimizing Matrix Transpose in CUDA

G. Ruetsch, P. Micikevicius - 2009

1 paper in library cites

Cited by

10

papers in your library

Cites

13

papers in your library

Read

on October 15, 2025

It's short, simple and straight to the point (as the network in the paper). It's refreshing to read something less "academic" and more "in your face"

Tags

Paper Aliases

No aliases