Papperoni

2015

Delving Deep Into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

K. He, X. Zhang, S. Ren, Jian Sun

Open PDF Google Scholar

citations

Cite Score

93

AI summary

The paper introduces Parametric Rectified Linear Unit (PReLU) for image classification, improving model fitting with negligible cost and little overfitting. It also introduces a robust initialization method for training extremely deep rectified models from scratch, achieving 4.94% top-5 error on the ImageNet 2012 dataset.

Main Contributions

Introduces Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit.
PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk.
Derives a robust initialization method that particularly considers the rectifier nonlinearities.
Enables training extremely deep rectified models directly from scratch.
Achieves 4.94% top-5 test error on the ImageNet 2012 classification dataset, surpassing human-level performance.

Abstract

Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66% [29]). To our knowledge, our result is the first to surpass human-level performance (5.1%, [22]) on this visual recognition challenge.

Citation Graph

Loading graph...

References [34]

Sort:

Filter:

[1]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[2]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[3]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[4]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[5]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[6]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[7]Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2010

18 papers in library cite

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

[8]The PASCAL Visual Object Classes (VOC) Challenge

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, Andrew Zisserman - 2010

7 papers in library cite

Good overview of the VOC challenge and good read, but too long and the part that they describe methods and results is very boring. Also, not related to NNs at the time!

[9]Visualizing and Understanding Convolutional Networks

Matthew D. Zeiler, Rob Fergus - 2014

15 papers in library cite

Very good explanation and visualization of CNNs, and also nice that they use their findings to improve the performance. The ablation study is also nice.

[10]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[11]Caffe: Convolutional Architecture for Fast Feature Embedding

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, Ross Girshick, S. Guadarrama, Trevor Darrell - 2014

12 papers in library cite

Nothing new really, but worth the read. It's nice because it's the precursor to current AI frameworks + has a Python interface. Also good that model representation is separate from implementation

[12]Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2014

6 papers in library cite

Very simple, general and effective method. The paper ends at page ~4 TBH, the rest is just results and gets boring. Good contribution though.

[13]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[14]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[15]Rectifier Nonlinearities Improve Neural Network Acoustic Models

A. L. Maas, A. Y. Hannun, Andrew Y. Ng - 2013

3 papers in library cite

It's a good paper, but I expected SO MUCH MORE! In the end it's like "eh, leaky relus train slightly faster" - uuughhh

[16]Network in Network

M. Lin, Qinlang Chen, Shuicheng Yan - 2013

11 papers in library cite

I think this was badly written and explained. The idea is nice but I didn't like the paper at all.

[17]Deepface: Closing the Gap to Human-Level Performance in Face Verification

Y. Taigman, Michael Yang, Marc'aurelio Ranzato, Lior Wolf - 2014

5 papers in library cite

Very impressive results but boring overall. I think the main thing is that they were the first to use CNN, but it seems like the most important part of their method was the 3D alignment (which is nice, but out of scope for me)

[18]Multi-Column Deep Neural Networks for Image Classification

Dan C. Ciresan, Ueli Meier, Jürgen Schmidhuber - 2012

11 papers in library cite

Very nice paper! And I am impressed they used CNNs before Hinton's paper. It's a shame there are so few citations. They also propose max-pooling and actually give a good explanation about it.

[19]Regularization of Neural Networks Using Dropconnect

L. Wan, M. Zeiler, S. Zhang, Rob Fergus - 2013

8 papers in library cite

I feel that the method is very complex and does not improve much on top of regular dropout.

[20]Deeply-Supervised Nets

Chen Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu - 2014

8 papers in library cite

Amazing potential, very nice idea, but bad writing.

[21]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[22]Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks

Surya Ganguli - 2014

9 papers in library cite

TBH it's been almost 2 months since I read this paper (shame on me for forgetting to add it)... Anyway, as I recall it I liked it, but TBH it's a bit underwhelming because it solved only for linear networks

[23]One Weird Trick for Parallelizing Convolutional Neural Networks

Alex Krizhevsky - 2014

3 papers in library cite

Very nice paper. Good new approach to training and explains the idea and rationale well. Not too many citations but used by SotA CNNs at the time.

[24]On Rectified Linear Units for Speech Processing

Matthew D. Zeiler, M. A. Ranzato, R. Monga, M. Mao, K. Yang, Quoc Le, P. Nguyen, A. Senior, Vincent Vanhoucke, Jeffrey Dean - 2013

3 papers in library cite

It's boring and the only thing it brings to the table is usage of ReLU in speech recog (which is unsurprising since it was already used everywhere else)

[25]OverFeat: Integrated Recognition, Localization and Detection Using Convolutional Networks

P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, Rob Fergus, Yann Lecun - 2014

16 papers in library cite

Very convoluted method, was SotA for only a bit of time, and the paper is very boring.

[26]Imagenet Large Scale Visual Recognition Challenge

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Zhongqiang Huang, A. Karpathy, A. Khosla, M. Bernstein - 2014

18 papers in library cite

Imagenet dataset challenge paper

[27]Return of the Devil in the Details: Delving Deep Into Convolutional Nets

K. Chatfield, K. Simonyan, A. Vedaldi, Andrew Zisserman - 2014

5 papers in library cite

Using CNNs as feature extractors

[28]Convolutional Neural Networks at Constrained Time Cost

K. He, Jian Sun - 2014

2 papers in library cite

Main arch for the "delving deep into RELU" paper (SotA on ILSVRC challenge)

[29]Some Improvements on Deep Convolutional Neural Network Based Image Classification

A. G. Howard - 2013

4 papers in library cite

CNNs with images of different scales

[30]Compete to Compute

R. K. Srivastava, Jonathan Masci, S. Kazerounian, Faustino Gomez, Jürgen Schmidhuber - 2013

3 papers in library cite

I like the name + schmidhuber

[31]Understanding Deep Architectures Using a Recursive Convolutional Network

D. Eigen, J. Rolfe, Rob Fergus, Yann Lecun - 2013

2 papers in library cite

"recent research on small datasets suggests that the accuracy should improve from the increased number of parameters in conv layers"

[32]Deep Image: Scaling Up Image Recognition

R. Wu, Y. Shan, G. Sun - 2015

2 papers in library cite

[33]Deep Learning Face Representation by Joint Identification-Verification

Y. S. Sun, Yanru Chen, Xinpeng Wang, X. Tang - 2014

1 paper in library cites

[34]Learning Activation Functions to Improve Deep Neural Networks

F. Agostinelli, M. Hoffman, P. Sadowski, P. Baldi - 2014

1 paper in library cites

Cited by

10

papers in your library

Cites

31

papers in your library

Read

on July 20, 2025

I think the PRELU idea didn't catch on, but the initialization is very nice! Good read.

Tags

Paper Aliases

No aliases