Papperoni

2017

Densely Connected Convolutional Networks

G. Huang, Ze Liu, K. Weinberger, Laurens Van Der Maaten

Open PDF Google Scholar

citations

Cite Score

97

AI summary

This paper introduces DenseNets, a novel convolutional network architecture that connects each layer to every other layer, addressing vanishing-gradient problems, strengthening feature propagation, encouraging feature reuse, and reducing parameters, achieving state-of-the-art results on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.

Main Contributions

Introduces Dense Convolutional Networks (DenseNets), a novel architecture connecting each layer to every other layer in a feed-forward fashion.
Alleviates the vanishing-gradient problem by strengthening feature propagation and encouraging feature reuse.
Reduces the number of parameters compared to traditional convolutional networks.
Achieves significant improvements over state-of-the-art results on CIFAR-10, CIFAR-100, SVHN, and ImageNet benchmark tasks.
Demonstrates improved information and gradient flow, making the networks easier to train.

Abstract

Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections—one between each layer and its subsequent layer—our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet.

Citation Graph

Loading graph...

References [42]

Sort:

Filter:

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[3]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[4]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[5]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[6]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[7]Fully Convolutional Networks for Semantic Segmentation

J. Long, E. Shelhamer, Trevor Darrell - 2015

7 papers in library cite

I didn't really like the way the paper is written and the results seem a bit underwhelming. However, it's nice that they do it in a fully convolutional way and that they increase the speed.

[8]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[9]Rethinking the Inception Architecture for Computer Vision

Zbigniew Wojna - 2015

5 papers in library cite

It's nice to see all of the performance optimizations they do, but it's very derivative

[10]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[11]Delving Deep Into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

K. He, X. Zhang, S. Ren, Jian Sun - 2015

10 papers in library cite

I think the PRELU idea didn't catch on, but the initialization is very nice! Good read.

[12]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[13]Identity Mappings in Deep Residual Networks

K. He, X. Zhang, S. Ren, Jian Sun - 2016

4 papers in library cite

This paper is amazing - simple, easy to follow, intuitive, while also being impactful! ResNets are already a big leap forward, and they can improve on that

[14]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[15]Network in Network

M. Lin, Qinlang Chen, Shuicheng Yan - 2013

11 papers in library cite

I think this was badly written and explained. The idea is nice but I didn't like the paper at all.

[16]Reading Digits in Natural Images With Unsupervised Feature Learning

Y. Netzer, Tianle Wang, A. Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng - 2011

8 papers in library cite

It's a bit meh. It's also 2011 so maybe it was impressive, but I think the main contribution is the dataset.

[17]Training Very Deep Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber - 2015

6 papers in library cite

The idea of using the same concepts of LSTM to feedforward nets is so simple yet it took so long! Very cool idea but the results are underwhelming (otherwise would be a 5!)

[18]Deeply-Supervised Nets

Chen Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu - 2014

8 papers in library cite

Amazing potential, very nice idea, but bad writing.

[19]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[20]Convolutional neural Networks Applied to House Numbers Digit Classification

P. Sermanet, S. Chintala, Yann Lecun - 2012

6 papers in library cite

This reads like an undergrad's research paper or a recent master's paper. Only applying stuff to a new dataset.

[21]Imagenet Large Scale Visual Recognition Challenge

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Zhongqiang Huang, A. Karpathy, A. Khosla, M. Bernstein - 2014

18 papers in library cite

Imagenet dataset challenge paper

[22]Wide Residual Networks

S. Zagoruyko, N. Komodakis - 2016

5 papers in library cite

Turn networks wide vs. deep

[23]Fitnets: Hints for Thin Deep Nets

A. Romero, Nicolas Ballas, S. E. Kahou, A. Chassang, C. Gatta, Yoshua Bengio - 2015

5 papers in library cite

Fitnets, referenced by resnets + bengio

[24]Deep Networks With Stochastic Depth

G. Huang, Y. S. Sun, Ze Liu, D. Sedra, K. Q. Weinberger - 2016

3 papers in library cite

'Stochastic Depth' - randomly dropping some layers

[25]The Cascade-Correlation Learning Architecture

S. E. Fahlman, C. Lebiere - 1989

6 papers in library cite

[26]Pedestrian Detection With Unsupervised Multi-Stage Feature Learning

P. Sermanet, Koray Kavukcuoglu, S. Chintala, Yann Lecun - 2013

4 papers in library cite

[27]Striving for Simplicity: The All Convolutional Net

J. T. Springenberg, Alexey Dosovitskiy, T. Brox, M. Riedmiller - 2014

4 papers in library cite

[28]Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

Q. Liao, T. Poggio - 2016

2 papers in library cite

[29]Hypercolumns for Object Segmentation and Fine-Grained Localization

B. Hariharan, P. Arbelaez, Ross Girshick, Jitendra Malik - 2015

2 papers in library cite

[30]A Neural Algorithm of Artistic Style

L. Gatys, A. Ecker, M. Bethge - 2015

1 paper in library cites

[31]Adanet: Adaptive Structural Learning of Artificial Neural Networks

C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, Shusheng Yang - 2016

1 paper in library cites

[32]Augmenting Supervised Neural Networks With Unsupervised Objectives for Large-Scale Image Classification

Y. Z. Zhang, K. Lee, Honglak Lee - 2016

1 paper in library cites

[33]Deconstructing the Ladder Network Architecture

M. Pezeshki, L. Fan, P. Brakel, Aaron Courville, Yoshua Bengio - 2016

1 paper in library cites

[34]Deep Manifold Traversal: Changing Labels With Convolutional Features

J. R. Gardner, M. J. Kusner, Yiwei Li, P. Upchurch, K. Q. Weinberger, J. E. Hopcroft - 2015

1 paper in library cites

[35]Deeply-Fused Nets

J. Wang, Z. Wei, Tong Zhang, W. Zeng - 2016

1 paper in library cites

[36]Fractalnet: Ultra-Deep Neural Networks Without Residuals

G. Larsson, M. Maire, G. Shakhnarovich - 2016

1 paper in library cites

[37]Memory-Efficient Implementation of Densenets

G. Pleiss, Deli Chen, G. Huang, Tao Li, Laurens Van Der Maaten, K. Q. Weinberger - 2017

1 paper in library cites

[38]Multi-Scale Recognition With Dag cnns

Shusheng Yang, D. Ramanan - 2015

1 paper in library cites

[39]Neural Network Learning Without Backpropagation

B. M. Wilamowski, H. Yu - 2010

1 paper in library cites

[40]Resnet in Resnet: Generalizing Residual Architectures

S. Targ, Diogo Almeida, K. Lyman - 2016

1 paper in library cites

[41]Semi-Supervised Learning With Ladder Networks

A. Rasmus, M. Berglund, M. Honkala, Harri Valpola, Tapani Raiko - 2015

1 paper in library cites

[42]Training and Investigating Residual Nets

S. Gross, M. Wilber - 2016

1 paper in library cites

Cited by

5

papers in your library

Cites

24

papers in your library

Read

on October 21, 2025

I liked this paper so much! The way that it's written makes it very easy to follow. Results are nice, explanations are intuitive. Very nice!

Tags

Paper Aliases

No aliases