Papperoni

2016

Deep Learning

I. Goodfellow, Yoshua Bengio, Y. A. Courville, A. Aaron

Open PDF Google Scholar

citations

Cite Score

99

AI summary

This paper reviews deep learning, representation-learning methods with multiple levels of representation, obtained by composing simple but non-linear modules. It surveys deep learning successes in image and speech recognition, natural language understanding, and other machine learning tasks. Finally, the paper highlights future research directions.

Main Contributions

Introduces deep learning as representation-learning methods with multiple levels of representation.
Presents a historical overview of deep learning, starting from the revival of interest in deep feedforward networks around 2006.
Surveys the successes of deep learning in image and speech recognition.
Reviews the applications of deep learning to natural language understanding, including neural language models and recurrent neural networks.
Discusses the future of deep learning, including the combination of representation learning with complex reasoning.

Abstract

Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer. Deep convolutional nets have brought about breakthroughs in processing images, video, speech and audio, whereas recurrent nets have shone light on sequential data such as text and speech.

Citation Graph

Loading graph...

References [103]

Sort:

Filter:

[1]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[2]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[3]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[4]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[5]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[6]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[7]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[8]Visualizing Data Using t-SNE

Geoffrey Hinton - 2008

7 papers in library cite

Amazing. Simple. Impactful. Easy to understand. Masterpiece.

[9]Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Ross Girshick, J. Donahue, Trevor Darrell, Jitendra Malik - 2014

18 papers in library cite

Good results, beat overfeat, used pretraining for improving performance. Only issue is that the paper is overly long...

[10]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[11]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[12]Human-Level Control Through Deep Reinforcement Learning

V. Mnih - 2015

9 papers in library cite

Maybe the first instance of RL with NNs! I am impressed that it seems quite straightforward to implement (maybe tricky to get working).

[13]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[14]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[15]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[16]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[17]Representation Learning: A Review and New Perspectives

Yoshua Bengio, Aaron Courville, A. A. Vincent, P. Pascal - 2013

4 papers in library cite

Good in spirit but is too long and covers a lot of things that are not relevant anymore.

[18]Deep Neural Networks for Acoustic Modeling in Speech Recognition

Geoffrey Hinton - 2012

21 papers in library cite

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

[19]Show, Attend and Tell: Neural Image Caption Generation With Visual Attention

K. Xu, Jimmy Lei Ba, R. Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, R. Zemel, Yoshua Bengio - 2015

12 papers in library cite

It's a nice paper. I liked the soft attention way more than the hard one, and I am a bit mad that it wasn't the best lol And also it's the first paper I read about multimodality, but it seems that this was bustling at the time. Also results are kinda bad.

[20]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[21]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[22]Learning Deep Architectures for AI

Yoshua Bengio - 2009

25 papers in library cite

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

[23]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[24]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[25]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[26]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[27]Deepface: Closing the Gap to Human-Level Performance in Face Verification

Y. Taigman, Michael Yang, Marc'aurelio Ranzato, Lior Wolf - 2014

5 papers in library cite

Very impressive results but boring overall. I think the main thing is that they were the first to use CNN, but it seems like the most important part of their method was the 3D alignment (which is nice, but out of scope for me)

[28]Show and Tell: A Neural Image Caption Generator

Dumitru Erhan - 2015

11 papers in library cite

It's nice and they beat a ton of SotA. However, I read the one that uses attention first so this is a bit less surprising.

[29]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[30]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[31]Handwritten Digit Recognition With a Back-Propagation Network

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel - 1990

10 papers in library cite

Just a minor derivation of the previous paper, adding more layers to the net. Pretty much the same exact text than the other one. Uninteresting if you read the other.

[32]Phoneme Recognition Using Time-Delay Neural Networks

A. H. Waibel, T. Hanazawa, Geoffrey Hinton, K. Shikano, K. Lang - 1989

13 papers in library cite

I didn't really like this paper, although it's a good early example of using backprop. I think it can be thought as a 1D convolution.

[33]Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

G. Dahl, D. Yu, L. Deng, Alex Acero - 2012

19 papers in library cite

Good paper, very well written and probably the best explanation of RBMs and DBNs I've seen. However, I don't see a lot of impact and seems very derivative from other works.

[34]Neural Turing Machines

Alex Graves, G. Wayne, Ivo Danihelka - 2014

18 papers in library cite

This paper is amazing. If someone told me that NNs could use and address memory by position I wouldn't believe it worked. Very nice, but it's a shame that it's just a toy example.

[35]Memory Networks

Jason Weston, S. Chopra, Antoine Bordes - 2015

18 papers in library cite

The first half of the paper (when they discuss the concept in a very abstract way) is amazing. However, the actual methodology was very convoluted - I did not like it. I thought that Neural Turing Machines were inspired in this, but actually they are contemporary... So anyway, the concept is nice, execution is not.

[36]Generating Text With Recurrent Neural Networks

Ilya Sutskever, James Martens, Geoffrey E. Hinton - 2011

13 papers in library cite

Pleasant paper but results are underwhelming. They use RNNs for character-level modeling, which is different. They also use the hessian-free method proposed by Martens, but don't go too deep into how it works, which is nice because otherwise it would be very mathy. Other papers cite this more as an example of usage rather than an actual milestone.

[37]Deep Convolutional Neural Networks for LVCSR

T. Sainath, Abdel Rahman Mohamed, Brian Kingsbury, Bhuvana Ramabhadran - 2013

2 papers in library cite

Nice to see CNNs for other things other than images, but I am biased against speech recognition and CNNs. Very good that the paper is short though.

[38]Efficient Learning of Sparse Representations With an Energy-Based Model

Marc'aurelio Ranzato, C. Poultney, S. Chopra, Yann Lecun - 2006

20 papers in library cite

It's ok. Not really good, but alright.

[39]Parsing Natural Scenes and Natural Language With Recursive Neural Networks

Richard Socher, C. C. Lin, C. Manning, Andrew Y. Ng - 2011

10 papers in library cite

Good idea and nice results, but not my thing right now.

[40]The Wake-Sleep Algorithm for Unsupervised Neural Networks

Geoffrey Hinton, Peter Dayan, B. Frey, R. Neal - 1995

9 papers in library cite

It's okay... I get the feeling that this is early autoencoders work, but the term still didn't exist. I don't think it adds nothing new though.

[41]Towards AI-complete Question Answering: A Set of Prerequisite Toy Tasks

Jason Weston, Antoine Bordes, S. Chopra, Tomas Mikolov - 2015

11 papers in library cite

It's a good idea and a nice read but the bad part is that most of the tasks are already easy.

[42]On Using Very Large Target Vocabulary for Neural Machine Translation

Yoshua Bengio - 2014

12 papers in library cite

It's nice, but it starts getting a bit into the realm of "yeah, that seems like a minor improvement". It's nice that they use the importance sampling stuff from the previous paper though - I thought it had completely vanished :)

[43]Strategies for Training Large Scale Neural Network Language Models

Tomas Mikolov, A. Deoras, D. Povey, Lukas Burget, Jan Cernocky - 2011

9 papers in library cite

Just builds on other things. Very minor suff in my opinion.

[44]Continuous Space Language Models

Holger Schwenk - 2007

12 papers in library cite

One more paper about speech recog. Nothing special really.

[45]Une Procédure D'Apprentissage pour Réseau à Seuil Asymétrique

Yann Lecun - 1985

5 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[46]OverFeat: Integrated Recognition, Localization and Detection Using Convolutional Networks

P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, Rob Fergus, Yann Lecun - 2014

16 papers in library cite

Very convoluted method, was SotA for only a bit of time, and the paper is very boring.

[47]Learning Hierarchical Features for Scene Labeling

Clement Farabet, C. Couprie, L. Najman, Yann Lecun - 2013

6 papers in library cite

Cited by deep learning paper (and 4 more)

[48]Deep Boltzmann Machines

Ruslan Salakhutdinov, Geoffrey E. Hinton - 2009

9 papers in library cite

Not sure what's the difference vs. DBNs

[49]On the Number of Linear Regions of Deep Neural Networks

G. F. Montufar, Razvan Pascanu, Kyunghyun Cho, Yoshua Bengio - 2014

3 papers in library cite

Sounds very mathy, but bengio and seems to challenge the idea that multilayer networks can approximate complex functions

[50]Multiple Object Recognition With Visual Attention

Jimmy Lei Ba, V. Mnih, Koray Kavukcuoglu - 2014

4 papers in library cite

Cited by DRAW

[51]Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences

P. Werbos - 1974

14 papers in library cite

Werbos PhD Thesis, introduced backprop

[52]Acoustic Modeling Using Deep Belief Networks

A. Mohamed, G. Dahl, Geoffrey Hinton - 2012

12 papers in library cite

[53]Untersuchungen zu dynamischen neuronalen netzen

Sepp Hochreiter - 1991

18 papers in library cite

[54]Pattern Classification and Scene Analysis

R. O. Duda, P. E. Hart - 1973

9 papers in library cite

[55]Receptive Fields, Binocular Interaction and Functional Architecture in the Cat's Visual Cortex

D. H. Hubel, T. N. Wiesel - 1962

8 papers in library cite

[56]Neocognitron: A New algorithm for Pattern Recognition Tolerant of Deformations and Shifts in Position

Kunihiko Fukushima, S. Miyake - 1982

7 papers in library cite

[57]The Curse of Highly Variable Functions for Local Kernel Machines

Yoshua Bengio, O. Delalleau, N. L. Roux - 2006

7 papers in library cite

[58]Hierarchical Recurrent Neural Networks for Long-Term Dependencies

S. Elhihi, Yoshua Bengio - 1996

6 papers in library cite

[59]Best Practices for Convolutional Neural Networks

Patrice Y. Simard, D. Steinkraus, John C. Platt - 2003

4 papers in library cite

[60]Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization

Yann N. Dauphin, Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio - 2014

4 papers in library cite

[61]Large-Scale Deep Unsupervised Learning Using Graphics Processors

Rajat Raina, A. Madhavan, Andrew Y. Ng - 2009

4 papers in library cite

[62]Original Approach for the Localisation of Objects in Images

R. Vaillant, C. Monrocq, Yann Lecun - 1994

4 papers in library cite

[63]Pedestrian Detection With Unsupervised Multi-Stage Feature Learning

P. Sermanet, Koray Kavukcuoglu, S. Chintala, Yann Lecun - 2013

4 papers in library cite

[64]Synergistic Face Detection and Pose Estimation With Energy-Based Models

M. Osadchy, Yann Lecun, M. Miller - 2007

4 papers in library cite

[65]The Loss Surfaces of Multilayer Networks

A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, Yann Lecun - 2015

4 papers in library cite

[66]Toward Automatic Phenotyping of Developing Embryos From Videos

F. Ning, D. Delhomme, Yann Lecun, F. Piano, Leon Bottou, P. Barbano - 2005

4 papers in library cite

[67]Convolutional Face Finder: A Neural Architecture for Fast and Robust Face Detection

C. Garcia, M. Delakis - 2004

3 papers in library cite

[68]Convolutional Networks Can Learn to Generate Affinity Graphs for Image Segmentation

S. C. Turaga, J. F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, H. S. Seung - 2010

3 papers in library cite

[69]Deep Generative Stochastic Networks Trainable by backprop

Yoshua Bengio, E. T. Laufer, Jason Yosinski - 2013

3 papers in library cite

[70]Face Recognition: A Convolutional Neural-Network Approach

S. Lawrence, C. Giles, A. Tsoi, A. Back - 1997

3 papers in library cite

[71]From Machine Learning to Machine Reasoning

Leon Bottou - 2011

3 papers in library cite

[72]Learning Convolutional Feature Hierarchies for Visual Recognition

Koray Kavukcuoglu - 2010

3 papers in library cite

[73]Learning Fast Approximations of Sparse Coding

K. Gregor, Yann Lecun - 2010

3 papers in library cite

[74]Learning Long-Range Vision for Autonomous Off-Road Driving

Raia Hadsell, P. Sermanet, M. Scoffier, A. Erkan, K. Kavackuoglu, U. Muller, Yann Lecun - 2009

3 papers in library cite

[75]Training Recurrent Neural Networks

Ilya Sutskever - 2012

3 papers in library cite

[76]Distributed Hierarchical Processing in the Primate cerebral Cortex

D. J. Felleman, D. C. V. Essen - 1991

2 papers in library cite

[77]Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation

J. Tompson, A. Jain, Yann Lecun, C. Bregler - 2014

2 papers in library cite

[78]Large-Scale FPGA-based Convolutional Networks

Clement Farabet, Yann Lecun, Koray Kavukcuoglu, E. Culurciello, B. Martini, P. Akselrod, S. Talay - 2011

2 papers in library cite

[79]Question Answering With Subgraph Embeddings

Antoine Bordes, S. Chopra, Jason Weston - 2014

2 papers in library cite

[80]The Tradeoffs of Large Scale Learning

Leon Bottou, O. Bousquet - 2008

2 papers in library cite

[81]When Does a Mixture of Products Contain a Product of Mixtures?

G. Montufar, J. Morton - 2014

2 papers in library cite

[82]An Analog Neural Network Processor With Programmable Topology

B. Boser, E. Sackinger, J. Bromley, Yann Lecun, Lawrence Jackel - 1991

1 paper in library cites

[83]Connectomic Reconstruction of the Inner Plexiform Layer in the Mouse Retina

M. Helmstaedter - 2013

1 paper in library cites

[84]Deep Learning of the Tissue-Regulated Splicing Code

M. K. Leung, H. Y. Xiong, L. J. Lee, B. J. Frey - 2014

1 paper in library cites

[85]Deep Neural Nets as a Method for Quantitative Structure-Activity Relationships

J. Ma, R. P. Sheridan, A. Liaw, George E. Dahl, V. Svetnik - 2015

1 paper in library cites

[86]Deep Neural Networks Rival the Representation of Primate It Cortex for Core Visual Object Recognition

C. F. Cadieu - 2014

1 paper in library cites

[87]Efficient Object Localization Using Convolutional Networks

J. Tompson, R. R. Goroshin, A. Jain, Y. Y. Lecun, C. C. Bregler - 2014

1 paper in library cites

[88]Experiments With Time Delay Networks and Dynamic Time Warping for Speaker Independent Isolated Digit Recognition

Leon Bottou, F. F. Soulie, P. Blanchet, J. Lienard - 1989

1 paper in library cites

[89]Higgs Boson Machine Learning Challenge

Kaggle - 2014

1 paper in library cites

[90]In Neural Information Processing Systems

S. Nowlan, J. Platt - 1995

1 paper in library cites

[91]Learning Logic Report TR-47

D. B. Parker - 1985

1 paper in library cites

[92]Learning With Kernels

B. Scholkopf, A. Smola - 2002

1 paper in library cites

[93]Metaphors We Live By

G. Lakoff, M. J. Johnson - 2008

1 paper in library cites

[94]Modeling Natural Images Using Gated MRFS

Marc'aurelio Ranzato, V. Mnih, J. M. Susskind, Geoffrey E. Hinton - 2013

1 paper in library cites

[95]Multi-Column Deep Neural Network for Traffic Sign Classification

Dan C. Ciresan, Ueli Meier, Jonathan Masci, Jürgen Schmidhuber - 2012

1 paper in library cites

[96]Online Particle Detection With Neural Networks Based on Topological Calorimetry Information

T. Ciodaro, D. Deva, J. D. Seixas, D. Damazio - 2012

1 paper in library cites

[97]Pandemonium: A Paradigm for Learning in Mechanisation of Thought Processes

Oliver G. Selfridge - 1958

1 paper in library cites

[98]Scene Parsing With Multiscale Feature Learning, Purity Trees, and Optimal Covers

Clement Farabet, C. Couprie, L. Najman, Yann Lecun - 2012

1 paper in library cites

[99]Semantic Cognition: A Parallel Distributed Processing Approach

T. T. Rogers, J. L. Mcclelland - 2004

1 paper in library cites

[100]Semi-Supervised Learning With Deep Generative Models

D. Kingma, D. Rezende, S. Mohamed, M. Welling - 2014

1 paper in library cites

[101]The Human Splicing Code Reveals New Insights Into the Genetic Determinants of Disease

H. Y. Xiong - 2015

1 paper in library cites

[102]The Perceptron - A Perceiving and Recognizing Automaton

F. Rosenblatt - 1957

1 paper in library cites

[103]What Kind of Graphical Model Is the Brain?

Geoffrey E. Hinton - 2005

1 paper in library cites

Cited by

5

papers in your library

Cites

52

papers in your library

Read

on October 18, 2025

It's an awesome review for people that are not familiar with deep learning, but it's very basic as well. Not too relevant given the context of other papers I read, but huge impact in terms of scientific communication.

Tags

Paper Aliases

No aliases