Papperoni

2012

Multi-Column Deep Neural Networks for Image Classification

Dan C. Ciresan, Ueli Meier, Jürgen Schmidhuber

Open PDF Google Scholar

citations

Cite Score

78

AI summary

This paper introduces a multi-column deep neural network (MCDNN) architecture for image classification using graphics cards for fast training, achieving near-human performance on MNIST and outperforming humans on a traffic sign recognition benchmark. The MCDNN improves state-of-the-art results on various image classification benchmarks.

Main Contributions

Introduces a multi-column deep neural network (MCDNN) architecture.
Achieves near-human performance on the MNIST handwriting benchmark.
Outperforms humans by a factor of two on a traffic sign recognition benchmark.
Improves the state-of-the-art on a plethora of common image classification benchmarks.
Demonstrates that unsupervised initialization/pretraining is not necessary for properly trained big and deep DNNs.

Abstract

Traditional methods of computer vision and machine learning cannot match human performance on tasks such as the recognition of handwritten digits or traffic signs. Our biologically plausible deep artificial neural network architectures can. Small (often minimal) receptive fields of convolutional winner-take-all neurons yield large network depth, resulting in roughly as many sparsely connected neural layers as found in mammals between retina and visual cortex. Only winner neurons are trained. Several deep neural columns become experts on inputs preprocessed in different ways; their predictions are averaged. Graphics cards allow for fast training. On the very competitive MNIST handwriting benchmark, our method is the first to achieve near-human performance. On a traffic sign recognition benchmark it outperforms humans by a factor of two. We also improve the state-of-the-art on a plethora of common image classification benchmarks.

Citation Graph

Loading graph...

References [40]

Sort:

Filter:

[1]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[2]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[3]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[4]Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position

Kunihiko Fukushima - 1980

8 papers in library cite

Good read! First conv net? :)

[5]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[6]Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis

John C. Platt - 2003

12 papers in library cite

Very good paper. Simple and pragmatic.

[7]Why Does Unsupervised Pre-Training Help Deep Learning?

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre Antoine Manzagol, Pascal Vincent, Samy Bengio - 2010

12 papers in library cite

Good paper, easy to follow, and brings some light to the pre-training stuff (layer-by-layer). I just wish it wasn't so long. It's a chore.

[8]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[9]Flexible, High Performance Convolutional Neural Networks for Image Classification

Dan C. Ciresan, Ueli Meier, Jonathan Masci, Luca M. Gambardella, Jürgen Schmidhuber - 2011

3 papers in library cite

Schmidhuber + small filters CNN

[10]Learning Methods for Generic Object Recognition With Invariance to Pose and Lighting

Yann Lecun, Fu Jie Huang, Leon Bottou - 2004

18 papers in library cite

Good paper, nice methodology for creating different images. However, I think that this was not too impactful... I don't see this being used a lot.

[11]Efficient Learning of Sparse Representations With an Energy-Based Model

Marc'aurelio Ranzato, C. Poultney, S. Chopra, Yann Lecun - 2006

20 papers in library cite

It's ok. Not really good, but alright.

[12]Unsupervised Learning of Invariant Feature Hierarchies With Applications to Object Recognition

Marc'aurelio Ranzato, F. Huang, Y. Boureau, Yann Lecun - 2007

8 papers in library cite

I thought that I had learned pretty much everything before 2015, but this paper proved me wrong. Amazing read and very nice methodology. Very simple as well!

[13]Performance and Scalability of gpu-based Convolutional Neural Networks

D. Strigl, K. Kofler, Stefan Podlipnig - 2010

1 paper in library cites

Very good paper, and now it seems that I finally understand how CUDA works and how it is used to run NNs!

[14]Large-Scale Object Recognition With CUDA-accelerated Hierarchical Neural Networks

R. Uetz, Sven Behnke - 2009

2 papers in library cite

It's not a bad paper, but it seems like they tried to put too much stuff in it - the CNN part is very forgettable and the CUDA part is not so well explained as other papers.

[15]Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition

Dan C. Ciresan, Ueli Meier, Luca M. Gambardella, Jürgen Schmidhuber - 2010

10 papers in library cite

It's short, simple and straight to the point (as the network in the paper). It's refreshing to read something less "academic" and more "in your face"

[16]Une procédure d'apprentissage pour Réseau a seuil Asymmetrique (A Learning Scheme for Asymmetric Threshold Networks)

Yann Lecun - 1985

4 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[17]Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences

P. Werbos - 1974

14 papers in library cite

Werbos PhD Thesis, introduced backprop

[18]Hierarchical Models of Object Recognition in Cortex

M. Riesenhuber, T. Poggio - 1999

8 papers in library cite

[19]Receptive Fields, Binocular Interaction and Functional Architecture in the Cat's Visual Cortex

D. H. Hubel, T. N. Wiesel - 1962

8 papers in library cite

[20]Receptive Fields of Single Neurons in the Cat's Striate Cortex

D. H. Hubel, T. N. Wiesel - 1959

6 papers in library cite

[21]Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure

Ruslan Salakhutdinov, Geoffrey Hinton - 2007

5 papers in library cite

[22]The Importance of Encoding Versus Training With Sparse Coding and Vector Quantization

A. Coates, Andrew Y. Ng - 2011

5 papers in library cite

[23]Robust Object Recognition With Cortex-Like Mechanisms

T. Serre, Lior Wolf, S. Bileschi, M. Riesenhuber, T. Poggio - 2007

4 papers in library cite

[24]Traffic Sign Recognition With Multi-Scale Convolutional Networks

P. Sermanet, Yann Lecun - 2011

4 papers in library cite

[25]Hierarchical Neural Networks for Image Interpretation

Sven Behnke - 2003

3 papers in library cite

[26]Self-Organization and Associative Memory

T. Kohonen - 1988

3 papers in library cite

[27]Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition

D. Scherer, A. Muller, Sven Behnke - 2010

2 papers in library cite

[28]The German Traffic Sign Recognition Benchmark: A Multi-Class Classification Competition

J. Stallkamp, M. Schlipsing, J. Salmen, C. Igel - 2011

2 papers in library cite

[29]An Implicit Segmentation-Based Method for Recognition of Handwritten Strings of Characters

P. R. Cavalin, A. D. S. B. Jr, F. Bortolozzi, R. Sabourin, L. E. S. D. Oliveira - 2006

1 paper in library cites

[30]Chinese Handwriting Recognition Contest

C. L. Liu, F. Yin, D. H. Wang, Q. F. Wang - 2010

1 paper in library cites

[31]Convolutional Neural Network Committees for Handwritten Character Classification

Dan C. Ciresan, Ueli Meier, Luca M. Gambardella, Jürgen Schmidhuber - 2011

1 paper in library cites

[32]Estimating Accurate Multi-Class Probabilities With Support Vector Machines

J. Milgram, M. Cheriet, R. Sabourin - 2005

1 paper in library cites

[33]How Patterned Neural Connections Can Be Set Up by Self-Organization

D. J. Willshaw, C. V. D. Malsburg - 1976

1 paper in library cites

[34]Learning Algorithms for Classification: A Comparison on Handwritten Digit Recognition

Yann Lecun, L. D. Jackel, Leon Bottou, C. Cortes, John S. Denker, H. Drucker, I. Guyon, U. A. Muller, E. Sackinger, Patrice Simard, V. Vapnik - 1995

1 paper in library cites

[35]NIST Special Database 19 - Handprinted Forms and Characters Database

P. J. Grother - 1995

1 paper in library cites

[36]Overfitting in the Selection of Classifier Ensembles: A Comparative Study Between Pso and Ga

E. M. D. Santos, L. S. Oliveira, R. Sabourin, P. Maupin - 2008

1 paper in library cites

[37]Parallel and Serial Neural Mechanisms for Visual Search in Macaque Area V4

N. P. Bichot, A. F. Rossi, R. Desimone - 2005

1 paper in library cites

[38]Supervised Learning of Fuzzy ARTMAP Neural Networks Through Particle Swarm Optimization

E. Granger, P. Henniges, R. Sabourin - 2007

1 paper in library cites

[39]Unconstrained Handwritten Character Recognition Using Metaclasses of Characters

A. L. Koerich, P. R. Kalva - 2005

1 paper in library cites

[40]Version 7.10.0 (R2010a)

Matlab - 2010

1 paper in library cites

Cited by

11

papers in your library

Cites

17

papers in your library

Read

on August 1, 2025

Very nice paper! And I am impressed they used CNNs before Hinton's paper. It's a shame there are so few citations. They also propose max-pooling and actually give a good explanation about it.

Tags

Paper Aliases

No aliases