Papperoni

2013

Generating Sequences With Recurrent Neural Networks

Alex Graves

Open PDF Google Scholar

citations

Cite Score

75

AI summary

This paper introduces a system using Long Short-Term Memory (LSTM) recurrent neural networks for generating complex sequences with long-range structure. It demonstrates applications in text generation using the Penn Treebank and Hutter Prize Wikipedia datasets, and handwriting synthesis using the IAM Online Handwriting Database, achieving realistic results.

Main Contributions

Demonstrates the use of LSTM recurrent neural networks for generating complex sequences with long-range structure.
Applies the approach to text generation, achieving competitive performance on the Penn Treebank and generating samples from the Hutter Prize Wikipedia dataset.
Extends the approach to handwriting synthesis, generating realistic cursive handwriting conditioned on text sequences using the IAM Online Handwriting Database.
Introduces a novel convolutional mechanism that allows a recurrent network to condition its predictions on an auxiliary annotation sequence, enabling handwriting synthesis.

Abstract

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

Citation Graph

Loading graph...

References [33]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[3]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[4]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[5]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[6]Sequence Transduction With Recurrent Neural Networks

Alex Graves - 2012

7 papers in library cite

Good contribution. Discusses transducing (converting one sequence to the other) without pre-defined alignment. I didn't really like it as it is too mathy and a bit hard to understand, and I think it was not too impactful.

[7]Generating Text With Recurrent Neural Networks

Ilya Sutskever, James Martens, Geoffrey E. Hinton - 2011

13 papers in library cite

Pleasant paper but results are underwhelming. They use RNNs for character-level modeling, which is different. They also use the hessian-free method proposed by Martens, but don't go too deep into how it works, which is nice because otherwise it would be very mathy. Other papers cite this more as an example of usage rather than an actual milestone.

[8]Offline Handwriting Recognition With Multidimensional Recurrent Neural Networks

Alex Graves, Jürgen Schmidhuber - 2009

5 papers in library cite

It's okay. The method is nice but a bit too convoluted.

[9]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[10]Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity

Ronald J. Williams, David Zipser - 1992

8 papers in library cite

Oof, this was a hard read. I had to force myself to finish. I think this is too theoretical and deals with RNNs that are not really used nowadays. I skimmed through the second half.

[11]Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Pascal Vincent - 2012

8 papers in library cite

I think the main contribution is using HF optimization very early on, but very boring use case and they mix RNNs with RBMs which makes this soooo complicated.

[12]A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

A. Mnih, Yee Whye Teh - 2012

5 papers in library cite

Too mathy and really did not have a lot of impact TBH

[13]Subword Language Modeling With Neural Networks

Tomas Mikolov, Ilya Sutskever, A. Deoras, H. S. Le, S. Kombrink, Jan Cernocky - 2012

7 papers in library cite

Doesn't add a ton of contribution other than saying that subword models can perform better than char-level models. Probably the most important thing here is early use of subwords.

[14]Framewise Phoneme Classification With Bidirectional LSTM and Other Neural Network Architectures

Alex Graves, Jürgen Schmidhuber - 2005

14 papers in library cite

Very nice paper! Simple, no bullshit. Just "hey, we have LSTM and we have BRNN, let's try to join it"

[15]Neural Networks for Pattern Recognition

C. M. Bishop - 1995

12 papers in library cite

Book, 38k citations

[16]A Practical Guide to Training Restricted boltzmann Machines

Geoffrey E. Hinton - 2010

4 papers in library cite

[17]Statistical Language Models Based on Neural Networks

Tomas Mikolov - 2012

17 papers in library cite

Mikolov's Thesis

[18]Learning Precise Timing With LSTM Recurrent Networks

F. Gers, N. Schraudolph, Jürgen Schmidhuber - 2002

9 papers in library cite

[19]Practical Variational Inference for Neural Networks

Alex Graves - 2011

8 papers in library cite

[20]The Recurrent Temporal Restricted Boltzmann Machine

Ilya Sutskever, Geoffrey Hinton, G. Taylor - 2008

5 papers in library cite

[21]An Analysis of Noise in Recurrent Neural Networks: Convergence and Generalization

K. C. Jim, C. L. Giles, B. G. Horne - 1996

4 papers in library cite

[22]Lecture 6.5 - Rmsprop: Divide the Gradient by a Running Average of Its Recent Magnitude

T. Tieleman, Geoffrey Hinton - 2012

4 papers in library cite

[23]The Human Knowledge Compression Contest

M. Hutter - 2012

4 papers in library cite

[24]Factored Conditional Restricted boltzmann Machines for Modeling Motion Style

Graham W. Taylor, Geoffrey E. Hinton - 2009

3 papers in library cite

[25]IAM-OnDB an on-Line English Sentence Database Acquired From Handwritten Text on a Whiteboard

M. Liwicki, H. Bunke - 2005

3 papers in library cite

Missing year

[26]A First Look at Music Composition Using LSTM Recurrent Neural Networks

D. Eck, Jürgen Schmidhuber

1 paper in library cites

[27]A Machine Learning Perspective on Predictive Coding With Paq

B. Knoll, N. D. Freitas - 2011

1 paper in library cites

[28]Better Generative Models for Sequential Data Problems: Bidirectional Recurrent Mixture Density Networks

M. Schuster - 1999

1 paper in library cites

[29]Data Compression Using Adaptive Coding and Partial String Matching

J. G. Cleary, Ian, I. H. Witten - 1984

1 paper in library cites

[30]Low-Rank Matrix Factorization for Deep Neural Network Training With High-Dimensional Output Targets

T. N. Sainath, A. Mohamed, Brian Kingsbury, Bhuvana Ramabhadran - 2013

1 paper in library cites

[31]Mixture Density Networks

C. Bishop - 1994

1 paper in library cites

[32]The Minimum Description Length Principle (Adaptive Computation and Machine Learning)

P. D. Grunwald - 2007

1 paper in library cites

[33]The Tagged LOB corpus User's Manual

S. Johansson, R. Atwell, R. Garside, G. Leech - 1986

1 paper in library cites

Cited by

27

papers in your library

Cites

16

papers in your library

Read

on June 20, 2025

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

Tags

Paper Aliases

No aliases