Papperoni

2016

Identity Mappings in Deep Residual Networks

K. He, X. Zhang, S. Ren, Jian Sun

Open PDF Google Scholar

citations

Cite Score

88

AI summary

This paper analyzes deep residual networks, demonstrating that identity mappings as skip connections and after-addition activation enable direct signal propagation. They propose a new residual unit improving training and generalization, achieving improved results on CIFAR-10/100 and ImageNet using very deep ResNets.

Main Contributions

Demonstrates the importance of identity mappings in deep residual networks for direct signal propagation.
Proposes a new residual unit that makes training easier and improves generalization.
Achieves improved results on CIFAR-10 with a 1001-layer ResNet (4.62% error).
Achieves improved results on CIFAR-100 with a 1001-layer ResNet.
Achieves improved results on ImageNet with a 200-layer ResNet.

Abstract

Deep residual networks [1] have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers.

Citation Graph

Loading graph...

References [23]

Sort:

Filter:

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[3]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[4]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[5]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[6]Microsoft COCO: Common Objects in Context

T. Y. Lin, M. Maire, S. Belongie, James Hays, Pietro Perona, D. Ramanan, Piotr Dollar, C. L. Zitnick - 2014

14 papers in library cite

I liked this paper a lot. It's a bit long and I was already a bit tired, but it was nice overall.

[7]Rethinking the Inception Architecture for Computer Vision

Zbigniew Wojna - 2015

5 papers in library cite

It's nice to see all of the performance optimizations they do, but it's very derivative

[8]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[9]Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2010

18 papers in library cite

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

[10]Delving Deep Into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

K. He, X. Zhang, S. Ren, Jian Sun - 2015

10 papers in library cite

I think the PRELU idea didn't catch on, but the initialization is very nice! Good read.

[11]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[12]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[13]Network in Network

M. Lin, Qinlang Chen, Shuicheng Yan - 2013

11 papers in library cite

I think this was badly written and explained. The idea is nice but I didn't like the paper at all.

[14]Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

D. A. Clevert, Thomas Unterthiner, Sepp Hochreiter - 2016

2 papers in library cite

Very nice paper. I am not sure how influential it is, but I like how they explain the whole idea and motivation behind the ELU. I also like that the math is contained in a single section, and they give you a final summary to help understand what the math means.

[15]Training Very Deep Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber - 2015

6 papers in library cite

The idea of using the same concepts of LSTM to feedforward nets is so simple yet it took so long! Very cool idea but the results are underwhelming (otherwise would be a 5!)

[16]Deeply-Supervised Nets

Chen Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu - 2014

8 papers in library cite

Amazing potential, very nice idea, but bad writing.

[17]Imagenet Large Scale Visual Recognition Challenge

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Zhongqiang Huang, A. Karpathy, A. Khosla, M. Bernstein - 2014

18 papers in library cite

Imagenet dataset challenge paper

[18]Inception-V4, Inception-Resnet and the Impact of Residual Connections on Learning

Christian Szegedy, S. Ioffe, Vincent Vanhoucke, A. A. Alemi - 2017

3 papers in library cite

[19]Fitnets: Hints for Thin Deep Nets

A. Romero, Nicolas Ballas, S. E. Kahou, A. Chassang, C. Gatta, Yoshua Bengio - 2015

5 papers in library cite

Fitnets, referenced by resnets + bengio

[20]Highway Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber - 2015

6 papers in library cite

Introduced highway networks, which seem like a precursor to resnets

[21]All You Need Is a Good Init

D. Mishkin, J. Matas - 2016

2 papers in library cite

Importance of initialization

[22]Striving for Simplicity: The All Convolutional Net

J. T. Springenberg, Alexey Dosovitskiy, T. Brox, M. Riedmiller - 2014

4 papers in library cite

[23]Fractional max-pooling

B. Graham - 2014

2 papers in library cite

Cited by

4

papers in your library

Cites

21

papers in your library

Read

on October 31, 2025

This paper is amazing - simple, easy to follow, intuitive, while also being impactful! ResNets are already a big leap forward, and they can improve on that

Tags

Paper Aliases

No aliases