Papperoni

2016

Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu

Open PDF Google Scholar

citations

Cite Score

47

AI summary

This paper introduces an extension of Recurrent Neural Networks (RNNs) for large-scale language modeling using character Convolutional Neural Networks or Long-Short Term Memory on the One Billion Word Benchmark, achieving a perplexity of 23.7 using model ensembling.

Main Contributions

Explored and extended current research on large scale language modeling
Designed a Softmax loss based on character level CNNs that is efficient and precise.
Improved the state-of-the-art on a well known, large scale LM task, reducing perplexity from 51.3 down to 30.0.
Demonstrated that an ensemble of different models can reduce perplexity to 23.7.
Shared the models and recipes to help and motivate further research.

Abstract

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.

Citation Graph

Loading graph...

References [53]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[4]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[5]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[6]TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

M. Abadi, Akshat Agarwal, P. Barham, E. Brevdo, Ziru Chen, C. Citro, G. Corrado, A. Davis, Jeffrey Dean, M. Devin, Sanjay Ghemawat, I. Goodfellow, A. Harp, Geoffrey Irving, M. Isard, Y. Jia, R. Jozefowicz, Lukasz Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, Christopher Olah, M. Schuster, J. Shlens, B. Steiner, Ilya Sutskever, K. Talwar, P. Tucker, Vincent Vanhoucke, V. Vasudevan, F. Viegas, Oriol Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, Xiaoqiang Zheng - 2015

11 papers in library cite

This should be the golden standard to what framework papers should be. It's large, but it's not boring at all. Explains the core concepts while not going too deep as to describe unimportant things; explains design decisions and shortcomings... overall amazing

[7]Bidirectional Recurrent Neural Networks

M. Schuster, Kuldip K. Paliwal - 1997

10 papers in library cite

Introduced the BRNN concept. Good paper.

[8]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[9]Learning to Forget: Continual Prediction With LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins - 2000

13 papers in library cite

This defined what an LSTM is. Before that the LSTM didn't know how to forget. Very nice.

[10]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[11]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[12]Handwritten Digit Recognition With a Back-Propagation Network

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel - 1990

10 papers in library cite

Just a minor derivation of the previous paper, adding more layers to the net. Pretty much the same exact text than the other one. Uninteresting if you read the other.

[13]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[14]A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner - 2014

7 papers in library cite

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

[15]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[16]A Neural Attention Model for Abstractive Sentence Summarization

Alexander M. Rush, S. Chopra, Jason Weston - 2015

13 papers in library cite

TBH the paper is a bit boring and nothing new after reading a bunch of more modern techniques. I feel that they could have done a better job considering that seq2seq existed at the time. Either way, points for being the first to propose summarization with NNs.

[17]Training Very Deep Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber - 2015

6 papers in library cite

The idea of using the same concepts of LSTM to feedforward nets is so simple yet it took so long! Very cool idea but the results are underwhelming (otherwise would be a 5!)

[18]A Neural Conversational Model

Oriol Vinyals, Quoc V. Le - 2015

7 papers in library cite

No new methodology, no measurements, no results. It should be like 4 pages long if they didn't fill in with the conversation logs.

[19]Improved Backing-Off for M-Gram language Modeling

R. Kneser, Hermann Ney - 1995

11 papers in library cite

It's nice, it's simple... But not NNs and seems very incremental on top of existing backoff

[20]Generating Text With Recurrent Neural Networks

Ilya Sutskever, James Martens, Geoffrey E. Hinton - 2011

13 papers in library cite

Pleasant paper but results are underwhelming. They use RNNs for character-level modeling, which is different. They also use the hessian-free method proposed by Martens, but don't go too deep into how it works, which is nice because otherwise it would be very mathy. Other papers cite this more as an example of usage rather than an actual milestone.

[21]Hierarchical Probabilistic Neural Network Language Model

F. Morin, Yoshua Bengio - 2005

19 papers in library cite

Nice paper overall. Seems very impactful despite not being as relevant right now.

[22]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[23]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[24]Addressing the Rare Word Problem in Neural Machine Translation

T. Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba - 2014

14 papers in library cite

The method was very poorly explained. It was also worse than a paper released sooner, and more complicated. Overall not that good.

[25]Context Dependent Recurrent Neural Network Language Model

Tomas Mikolov, Geoffrey Zweig - 2012

12 papers in library cite

Nothing too interesting, just using the context of the RNN.

[26]Empirical Evaluation and Combination of Advanced Language Modeling Techniques

Tomas Mikolov, A. Deoras, S. Kombrink, Lukas Burget, Jan Cernocky - 2011

13 papers in library cite

Early work proving that NNs can be good. But very uninteresting overall.

[27]Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model

Yoshua Bengio, Jean Sebastien Senecal - 2008

6 papers in library cite

Just a rerun of a previous paper. This is ok, but really not too different.

[28]Multilingual Language Processing From Bytes

Amarnag Subramanya - 2016

2 papers in library cite

Nothing impressive... Their main innovation was the byte stuff. Plus the old-school NLP stuff makes it boring.

[29]Quick Training of Probabilistic Neural Nets by Importance Sampling

Yoshua Bengio, Jean Sebastien Senecal - 2003

11 papers in library cite

Good idea to overcome softmax computation cost. Not sure if too relevant today, but definitely better than the 2008 paper that is the same stuff.

[30]Framewise Phoneme Classification With Bidirectional LSTM and Other Neural Network Architectures

Alex Graves, Jürgen Schmidhuber - 2005

14 papers in library cite

Very nice paper! Simple, no bullshit. Just "hey, we have LSTM and we have BRNN, let's try to join it"

[31]Unsupervised Learning of Video Representations Using LSTMS

N. Srivastava, E. Mansimov, Ruslan Salakhutdinov - 2015

3 papers in library cite

Video representation

[32]Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling

H. Sak, A. W. Senior, F. Beaufays - 2014

5 papers in library cite

Only 5 pages

[33]Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation

Holger Schwenk, A. Rousseau, M. Attik - 2012

5 papers in library cite

[34]Statistical Language Models Based on Neural Networks

Tomas Mikolov - 2012

17 papers in library cite

Mikolov's Thesis

[35]Character-Aware Neural Language Models

Yoon Kim, Yacine Jernite, D. Sontag, Alexander M. Rush - 2016

7 papers in library cite

[36]Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models

M. Gutmann, A. Hyvarinen - 2010

7 papers in library cite

[37]Improving Neural Networks With Dropout

N. Srivastava - 2013

6 papers in library cite

[38]An Efficient Gradient-Based Algorithm for on-Line Training of Recurrent Network Trajectories

Ronald J. Williams, J. Peng - 1990

5 papers in library cite

[39]Decoding With Large-Scale Neural Language Models Improves Translation

Ashish Vaswani, Y. Zhao, V. Fossum, D. Chiang - 2013

5 papers in library cite

[40]Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

W. Ling, C. Dyer, A. W. Black, I. Trancoso, R. Fermandez, S. Amir, L. Marujo, T. Luis - 2015

5 papers in library cite

[41]An Empirical Exploration of Recurrent Network Architectures

R. Jozefowicz, Wojciech Zaremba, Ilya Sutskever - 2015

4 papers in library cite

[42]Larger-Context Language Modelling

Tianle Wang, Kyunghyun Cho - 2015

4 papers in library cite

[43]Learning Word Embeddings Efficiently With Noise-Contrastive Estimation

A. Mnih, Koray Kavukcuoglu - 2013

4 papers in library cite

[44]Document Context Language Models

Yangfeng Ji, T. Cohn, L. Kong, C. Dyer, J. Eisenstein - 2015

3 papers in library cite

[45]Improved Transition-Based Parsing by Modeling Characters Instead of Words With LSTMs

M. Ballesteros, C. Dyer, Noah A. Smith - 2015

3 papers in library cite

[46]Neural Probabilistic Language Models

Yoshua Bengio, Holger Schwenk, Jean Sebastien Senecal, F. Morin, Jean Luc Gauvain - 2006

3 papers in library cite

[47]Sparse Non-Negative Matrix Language Modeling for Skip-Grams

Noam Shazeer, J. Pelemans, C. Chelba - 2015

3 papers in library cite

[48]Blackout: Speeding Up Recurrent Neural Network Language Models With Very Large Vocabularies

S. Ji, S. V. N. Vishwanathan, S. Nadathur, M. J. Anderson, P. Dubey - 2015

2 papers in library cite

[49]Deep Neural Network Language Models

E. Arisoy, T. N. Sainath, Brian Kingsbury, Bhuvana Ramabhadran - 2012

2 papers in library cite

[50]Efficient Exact Gradient Update for Training Deep Networks With Very Large Sparse Targets

Pascal Vincent, A. D. Brebisson, X. Bouthillier - 2015

2 papers in library cite

[51]Hierarchical Neural Network Generative Models for Movie Dialogues

I. V. Serban, A. Sordoni, Yoshua Bengio, Aaron Courville, J. Pineau - 2015

1 paper in library cites

[52]Scaling Recurrent Neural Network Language Models

W. Williams, N. Prasad, D. Mrva, T. Ash, Tony Robinson - 2015

1 paper in library cites

[53]Sentence Compression by Deletion With Istms

K. Filippova, E. Alfonseca, C. A. Colmenares, Lukasz Kaiser, Oriol Vinyals - 2015

1 paper in library cites

Cited by

20

papers in your library

Cites

32

papers in your library

Read

on August 18, 2025

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

Tags

Paper Aliases

No aliases