Papperoni

2013

How to Construct Deep Recurrent Neural Networks

Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

45

AI summary

This paper explores how to extend RNNs to deep RNNs by analyzing three points: input-to-hidden, hidden-to-hidden transition, and hidden-to-output functions. It introduces two novel deep RNN architectures and a neural operator framework, evaluated on polyphonic music prediction and language modeling, demonstrating performance gains.

Main Contributions

Identified three points in RNN architecture that can be deepened: input-to-hidden, hidden-to-hidden, and hidden-to-output functions.
Proposed two novel deep RNN architectures orthogonal to stacking recurrent layers.
Introduced a novel framework based on neural operators for interpreting deep RNNs.
Empirically evaluated the proposed deep RNNs on polyphonic music prediction and language modeling tasks.
Demonstrated that proposed deep RNNs benefit from depth and outperform conventional shallow RNNs.

Abstract

In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

Citation Graph

Loading graph...

References [46]

Sort:

Filter:

[1]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[2]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[3]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[4]Multilayer Feedforward Networks Are Universal Approximators

Kur Hornik, Maxwell Stinchcombe, Halbert White - 1989

3 papers in library cite

It's a bunch of proofs, so not nice to read but it's very impactful.

[5]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[6]Deep Neural Networks for Acoustic Modeling in Speech Recognition

Geoffrey Hinton - 2012

21 papers in library cite

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

[7]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[8]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[9]Learning Deep Architectures for AI

Yoshua Bengio - 2009

25 papers in library cite

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

[10]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[11]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[12]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[13]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[14]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[15]On the Importance of Initialization and Momentum in Deep Learning

Ilya Sutskever, James Martens, G. Dahl, Geoffrey Hinton - 2013

13 papers in library cite

They give very good context and it's easy to understand that they are doing this as a counterpoint to HF. Surprising results as well. I just think it was made obsolete by relu

[16]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[17]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[18]Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

3 papers in library cite

Nothing surprising really, just domain adaptation using NNs.

[19]Generating Text With Recurrent Neural Networks

Ilya Sutskever, James Martens, Geoffrey E. Hinton - 2011

13 papers in library cite

Pleasant paper but results are underwhelming. They use RNNs for character-level modeling, which is different. They also use the hessian-free method proposed by Martens, but don't go too deep into how it works, which is nice because otherwise it would be very mathy. Other papers cite this more as an example of usage rather than an actual milestone.

[20]Extensions of Recurrent Neural Network Language Model

Tomas Mikolov, S. Kombrink, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2011

16 papers in library cite

Doesn't add much.

[21]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[22]Deep Learning Made Easier by Linear Transformations in Perceptrons

Tapani Raiko, Harri Valpola, Yann Lecun - 2012

7 papers in library cite

Kudos for introducing shortcut connections (which would become important in the future), but to me it seems a bit mid.

[23]Deep Learning via Hessian-Free Optimization

James Martens - 2010

12 papers in library cite

This paper is surprisingly good! When I first read the Hessian-Free optimization part, I thought "ugh, this is going to be full of math", but in the end it was very very enjoyable. I think I just wouldn't give it a 5 because it doesn't seem to have had that much impact.

[24]Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Pascal Vincent - 2012

8 papers in library cite

I think the main contribution is using HF optimization very early on, but very boring use case and they mix RNNs with RBMs which makes this soooo complicated.

[25]Learning Recurrent Neural Networks With Hessian-Free Optimization

James Martens, Ilya Sutskever - 2011

13 papers in library cite

Meh, very very mathy and seems like a minor improvement on LSTMs. They give very impressive results but I don't think it's too much of a fair comparison (vs. a 1999 architecture of LSTMs). I think they did well in showing that exploding/vanishing gradients can be overcome, but their method is not "it". As said by Bengio, they are partially responsible for the revival of RNNs (alongside Mikolov)

[26]Measuring Invariances in Deep Networks

I. Goodfellow, Quoc Le, A. Saxe, A. Ng - 2009

7 papers in library cite

Very nice concept and methodology, but the results in the end are underwhelming

[27]Subword Language Modeling With Neural Networks

Tomas Mikolov, Ilya Sutskever, A. Deoras, H. S. Le, S. Kombrink, Jan Cernocky - 2012

7 papers in library cite

Doesn't add a ton of contribution other than saying that subword models can perform better than char-level models. Probably the most important thing here is early use of subwords.

[28]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[29]Statistical Language Models Based on Neural Networks

Tomas Mikolov - 2012

17 papers in library cite

Mikolov's Thesis

[30]Learning Complex, Extended Sequences Using the Principle of History Compression

Jürgen Schmidhuber - 1992

8 papers in library cite

[31]Practical Variational Inference for Neural Networks

Alex Graves - 2011

8 papers in library cite

[32]Hierarchical Recurrent Neural Networks for Long-Term Dependencies

S. Elhihi, Yoshua Bengio - 1996

6 papers in library cite

[33]A Novel Connectionist System for Unconstrained Handwriting Recognition

Alex Graves, M. Liwicki, Santiago Fernandez, R. Bertolami, H. Bunke, Jürgen Schmidhuber - 2009

5 papers in library cite

[34]The Neural Autoregressive Distribution Estimator

Hugo Larochelle, I. Murray - 2011

5 papers in library cite

[35]Better Mixing via Deep Representations

Yoshua Bengio, G. Mesnil, Yann Dauphin, S. Rifai - 2013

3 papers in library cite

[36]On Fast Dropout and Its Applicability to Recurrent Networks

J. Bayer - 2013

3 papers in library cite

[37]Revisiting Natural Gradient for Deep Networks

Razvan Pascanu, Yoshua Bengio - 2013

3 papers in library cite

[38]Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks

C. G. Gulcehre, Kyunghyun Cho, Razvan Pascanu, Yoshua Bengio - 2013

2 papers in library cite

[39]Recurrent Convolutional Neural Networks for Scene Labeling

P. Pinheiro, Ronan Collobert - 2014

2 papers in library cite

[40]Shallow vs. Deep Sum-Product Networks

O. Delalleau, Yoshua Bengio - 2011

2 papers in library cite

[41]Training and Analysing Deep Recurrent Neural Networks

M. Hermans, B. Schrauwen - 2013

2 papers in library cite

[42]A New Method for Learning Deep Recurrent Neural Networks

Jixuan Chen, L. Deng - 2013

1 paper in library cites

[43]An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models

Harri Valpola, J. Karhunen - 2002

1 paper in library cites

[44]Discovering Multiscale Dynamical Features With Hierarchical Echo State Networks

Herbert Jaeger - 2007

1 paper in library cites

[45]Gp-Bayesfilters: Bayesian Filtering Using Gaussian Process Prediction and Observation Models

J. Ko, F. Dieter - 2009

1 paper in library cites

[46]On the Number of Response Regions of Deep Feed Forward Networks With Piece-Wise Linear Activations

Razvan Pascanu, G. Montufar, Yoshua Bengio - 2013

1 paper in library cites

Cited by

7

papers in your library

Cites

28

papers in your library

Read

on June 7, 2025

Very interesting despite not being too relevant. Good read and a new way of thinking about RNNs.

Tags

Paper Aliases

No aliases