Papperoni

2012

ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton

Open PDF Google Scholar

citations

Cite Score

99

AI summary

This paper introduces a deep convolutional neural network trained on ImageNet LSVRC-2010 dataset, achieving top-1 and top-5 error rates of 37.5% and 17.0%, respectively, using novel ReLU nonlinearity, multi-GPU training, local response normalization, and overlapping pooling.

Main Contributions

Introduced a large, deep convolutional neural network for ImageNet classification.
Achieved state-of-the-art results on ImageNet LSVRC-2010 and ILSVRC-2012 competitions.
Developed a highly-optimized GPU implementation of 2D convolution.
Employed ReLU nonlinearity for faster training.
Utilized dropout for regularization and prevention of overfitting.

Abstract

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

Citation Graph

Loading graph...

References [26]

Sort:

Filter:

[1]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[2]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[3]Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2010

18 papers in library cite

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

[4]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[5]Handwritten Digit Recognition With a Back-Propagation Network

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel - 1990

10 papers in library cite

Just a minor derivation of the previous paper, adding more layers to the net. Pretty much the same exact text than the other one. Uninteresting if you read the other.

[6]Multi-Column Deep Neural Networks for Image Classification

Dan C. Ciresan, Ueli Meier, Jürgen Schmidhuber - 2012

11 papers in library cite

Very nice paper! And I am impressed they used CNNs before Hinton's paper. It's a shame there are so few citations. They also propose max-pooling and actually give a good explanation about it.

[7]Learning Generative Visual Models From Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories

Li Fei Fei, Rob Fergus, Pietro Perona - 2004

15 papers in library cite

I think most people cite this thinking this is where the Caltech 101 dataset comes from (it's not). Anyway, it's just an extension of the other dataset and it's very mathy, not NNs, and uninteresting.

[8]Labelme: A Database and Web-Based Tool for Image Annotation

Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, William T. Freeman - 2008

10 papers in library cite

It's a good paper overall but not worth the read. They just describe the platform (which for the time may have been a paradigm shift from in-house datasets). Maybe the basis for Amazon MT?

[9]Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis

John C. Platt - 2003

12 papers in library cite

Very good paper. Simple and pragmatic.

[10]What Is the Best Multi-Stage Architecture for Object Recognition?

K. Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun - 2009

20 papers in library cite

So boring and convoluted... They try to make things more abstract by saying "filter banks" and "2 stages", but in the end it's CNNs. It seems like they are trying to tie NNs to the rest of ML, but it doesn't work.

[11]Caltech-256 Object Category Dataset

Greg Griffin, Alex Holub, Pietro Perona - 2007

9 papers in library cite

Boring read, but at least they were somewhat impactful.

[12]Learning Methods for Generic Object Recognition With Invariance to Pose and Lighting

Yann Lecun, Fu Jie Huang, Leon Bottou - 2004

18 papers in library cite

Good paper, nice methodology for creating different images. However, I think that this was not too impactful... I don't see this being used a lot.

[13]High-Performance Neural Networks for Visual Object Classification

Dan C. Ciresan, Ueli Meier, Jonathan Masci, Luca M. Gambardella, Jürgen Schmidhuber - 2011

5 papers in library cite

OK, I can see why this is important (implementing CNNs in GPUs before it was cool), but nothing too surprising. BTW, I read the 6 page version but this one has color so I kept this one.

[14]Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations

Honglak Lee, R. Grosse, R. Ranganath, Andrew Y. Ng - 2009

12 papers in library cite

[15]Random Forests

L. Breiman - 2001

5 papers in library cite

[16]Why Is Real-World Visual Object Recognition Hard?

N. Pinto, D. D. Cox, J. J. Dicarlo - 2008

5 papers in library cite

[17]High-Dimensional Signature Compression for Large-Scale Image Classification

J. Sanchez, F. Perronnin - 2011

4 papers in library cite

[18]A High-Throughput Screening Approach to Discovering Good Forms of Biologically Inspired Visual Representation

N. Pinto, D. Doukhan, J. J. Dicarlo, D. D. Cox - 2009

3 papers in library cite

[19]Convolutional Deep Belief Networks on cifar-10

Alex Krizhevsky - 2010

3 papers in library cite

[20]Convolutional Networks Can Learn to Generate Affinity Graphs for Image Segmentation

S. C. Turaga, J. F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, H. S. Seung - 2010

3 papers in library cite

[21]Convolutional Networks and Applications in Vision

Yann Lecun, Koray Kavukcuoglu, Clement Farabet - 2010

1 paper in library cites

[22]ILSVRC-2012

J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, Li Fei Fei - 2012

1 paper in library cites

[23]Large Scale Visual Recognition Challenge 2010

A. Berg, J. Deng, Li Fei Fei - 2010

1 paper in library cites

[24]Lessons From the netflix Prize Challenge

R. M. Bell, Y. Koren - 2007

1 paper in library cites

[25]Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost

T. Mensink, J. Verbeek, F. Perronnin, G. Csurka - 2012

1 paper in library cites

[26]Using Very Deep Autoencoders for Content-Based Image Retrieval

Alex Krizhevsky, Geoffrey E. Hinton - 2011

1 paper in library cites

Cited by

71

papers in your library

Cites

14

papers in your library

Read

on July 11, 2025

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

Tags

Paper Aliases

No aliases