Papperoni

2015

Training Very Deep Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber

Open PDF Google Scholar

citations

Cite Score

68

AI summary

This paper introduces highway networks, a novel architecture inspired by LSTMs, that allows unimpeded information flow across many layers using adaptive gating units. This architecture enables the training of very deep networks using simple gradient descent, achieving good results on MNIST and CIFAR datasets.

Main Contributions

Introduces highway networks, a new architecture for training very deep networks.
Uses an LSTM-inspired adaptive gating mechanism to allow information flow across layers.
Shows that highway networks can be trained directly through simple gradient descent, even with hundreds of layers.
Demonstrates that highway networks achieve good generalization performance on MNIST and CIFAR datasets.
Provides an analysis of the behavior of highway networks, including the routing of information and the importance of individual layers.

Abstract

Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.

Citation Graph

Loading graph...

References [38]

Sort:

Filter:

[1]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[2]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[3]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[4]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[5]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[6]Delving Deep Into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

K. He, X. Zhang, S. Ren, Jian Sun - 2015

10 papers in library cite

I think the PRELU idea didn't catch on, but the initialization is very nice! Good read.

[7]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[8]Caffe: Convolutional Architecture for Fast Feature Embedding

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, Ross Girshick, S. Guadarrama, Trevor Darrell - 2014

12 papers in library cite

Nothing new really, but worth the read. It's nice because it's the precursor to current AI frameworks + has a Python interface. Also good that model representation is separate from implementation

[9]Network in Network

M. Lin, Qinlang Chen, Shuicheng Yan - 2013

11 papers in library cite

I think this was badly written and explained. The idea is nice but I didn't like the paper at all.

[10]Learning to Forget: Continual Prediction With LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins - 2000

13 papers in library cite

This defined what an LSTM is. Before that the LSTM didn't know how to forget. Very nice.

[11]On the Importance of Initialization and Momentum in Deep Learning

Ilya Sutskever, James Martens, G. Dahl, Geoffrey Hinton - 2013

13 papers in library cite

They give very good context and it's easy to understand that they are doing this as a counterpoint to HF. Surprising results as well. I just think it was made obsolete by relu

[12]Multi-Column Deep Neural Networks for Image Classification

Dan C. Ciresan, Ueli Meier, Jürgen Schmidhuber - 2012

11 papers in library cite

Very nice paper! And I am impressed they used CNNs before Hinton's paper. It's a shame there are so few citations. They also propose max-pooling and actually give a good explanation about it.

[13]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[14]Deeply-Supervised Nets

Chen Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu - 2014

8 papers in library cite

Amazing potential, very nice idea, but bad writing.

[15]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[16]Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks

Surya Ganguli - 2014

9 papers in library cite

TBH it's been almost 2 months since I read this paper (shame on me for forgetting to add it)... Anyway, as I recall it I liked it, but TBH it's a bit underwhelming because it solved only for linear networks

[17]Flexible, High Performance Convolutional Neural Networks for Image Classification

Dan C. Ciresan, Ueli Meier, Jonathan Masci, Luca M. Gambardella, Jürgen Schmidhuber - 2011

3 papers in library cite

Schmidhuber + small filters CNN

[18]Deep Learning Made Easier by Linear Transformations in Perceptrons

Tapani Raiko, Harri Valpola, Yann Lecun - 2012

7 papers in library cite

Kudos for introducing shortcut connections (which would become important in the future), but to me it seems a bit mid.

[19]Learning Recurrent Neural Networks With Hessian-Free Optimization

James Martens, Ilya Sutskever - 2011

13 papers in library cite

Meh, very very mathy and seems like a minor improvement on LSTMs. They give very impressive results but I don't think it's too much of a fair comparison (vs. a 1999 architecture of LSTMs). I think they did well in showing that exploding/vanishing gradients can be overcome, but their method is not "it". As said by Bengio, they are partially responsible for the revival of RNNs (alongside Mikolov)

[20]Fitnets: Hints for Thin Deep Nets

A. Romero, Nicolas Ballas, S. E. Kahou, A. Chassang, C. Gatta, Yoshua Bengio - 2015

5 papers in library cite

Fitnets, referenced by resnets + bengio

[21]On the Number of Linear Regions of Deep Neural Networks

G. F. Montufar, Razvan Pascanu, Kyunghyun Cho, Yoshua Bengio - 2014

3 papers in library cite

Sounds very mathy, but bengio and seems to challenge the idea that multilayer networks can approximate complex functions

[22]Highway Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber - 2015

6 papers in library cite

Introduced highway networks, which seem like a precursor to resnets

[23]Grid Long Short-Term Memory

N. Kalchbrenner, Ivo Danihelka, Alex Graves - 2016

3 papers in library cite

[24]Compete to Compute

R. K. Srivastava, Jonathan Masci, S. Kazerounian, Faustino Gomez, Jürgen Schmidhuber - 2013

3 papers in library cite

I like the name + schmidhuber

[25]Untersuchungen zu dynamischen neuronalen netzen

Sepp Hochreiter - 1991

18 papers in library cite

[26]Learning Complex, Extended Sequences Using the Principle of History Compression

Jürgen Schmidhuber - 1992

8 papers in library cite

[27]On the Power of Small-Depth Threshold Circuits

J. Hastad, M. Goldmann - 1991

7 papers in library cite

[28]Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization

Yann N. Dauphin, Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio - 2014

4 papers in library cite

[29]Striving for Simplicity: The All Convolutional Net

J. T. Springenberg, Alexey Dosovitskiy, T. Brox, M. Riedmiller - 2014

4 papers in library cite

[30]Bridging Long Time Lags by Weight Guessing and Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1996

3 papers in library cite

[31]Computational Limitations of Small-Depth Circuits

J. Hastad - 1987

3 papers in library cite

[32]Feature Learning in Deep Neural Networks-Studies on Speech Recognition Tasks

D. Yu, M. L. Seltzer, Jeffrey Li, J. T. Huang, F. Seide - 2013

2 papers in library cite

[33]Spatially-Sparse Convolutional Neural Networks

B. Graham - 2014

2 papers in library cite

[34]Deep Networks With Internal Selective Attention Through Feedback Connections

M. F. Stollenga, Jonathan Masci, Faustino Gomez, Jürgen Schmidhuber - 2014

1 paper in library cites

[35]On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures

M. Bianchini, F. Scarselli - 2014

1 paper in library cites

[36]On the Expressive Efficiency of Sum Product Networks

James Martens, V. Medabalimi - 2014

1 paper in library cites

[37]Random Walk Initialization for Training Very Deep Feedforward Networks

D. Sussillo, L. F. Abbott - 2014

1 paper in library cites

[38]Understanding Locally Competitive Networks

R. K. Srivastava, Jonathan Masci, Faustino Gomez, Jürgen Schmidhuber - 2015

1 paper in library cites

Cited by

6

papers in your library

Cites

24

papers in your library

Read

on December 2, 2025

The idea of using the same concepts of LSTM to feedforward nets is so simple yet it took so long! Very cool idea but the results are underwhelming (otherwise would be a 5!)

Tags

Paper Aliases

No aliases