Papperoni

2012

Context Dependent Recurrent Neural Network Language Model

Tomas Mikolov, Geoffrey Zweig

Open PDF Google Scholar

citations

Cite Score

32

AI summary

This paper introduces a context-dependent recurrent neural network language model (RNNLM) that incorporates contextual information using Latent Dirichlet Allocation (LDA), achieving state-of-the-art perplexity on the Penn Treebank and improvements in word-error-rate on the Wall Street Journal task.

Main Contributions

Introduced the use of context vectors to improve the performance of a RNNLM.
Demonstrated perplexity improvements over the previous state-of-the-art for the Penn Treebank.
Developed an efficient method for computing context vectors when using a sliding window of context.
Evaluated the models by rescoring N-best lists from a speech recognizer and observe improvements there as well.
Achieved WER improvements for the Wall Street Journal task.

Abstract

Recurrent neural network language models (RNNLMs) have recently demonstrated state-of-the-art performance across a variety of tasks. In this paper, we improve their performance by providing a contextual real-valued input vector in association with each word. This vector is used to convey contextual information about the sentence being modeled. By performing Latent Dirichlet Allocation using a block of preceding text, we achieve a topic-conditioned RNNLM. This approach has the key advantage of avoiding the data fragmentation associated with building multiple topic models on different data subsets. We report perplexity results on the Penn Treebank data, where we achieve a new state-of-the-art. We further apply the model to the Wall Street Journal speech recognition task, where we observe improvements in word-error-rate.

Citation Graph

Loading graph...

References [37]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[3]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[4]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[5]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[6]Extensions of Recurrent Neural Network Language Model

Tomas Mikolov, S. Kombrink, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2011

16 papers in library cite

Doesn't add much.

[7]Three New Graphical Models for Statistical Language Modelling

A. Mnih, Geoffrey Hinton - 2007

12 papers in library cite

I don't know why this is so impactful. I didn't like it and I think this was overly complex.

[8]Learning Recurrent Neural Networks With Hessian-Free Optimization

James Martens, Ilya Sutskever - 2011

13 papers in library cite

Meh, very very mathy and seems like a minor improvement on LSTMs. They give very impressive results but I don't think it's too much of a fair comparison (vs. a 1999 architecture of LSTMs). I think they did well in showing that exploding/vanishing gradients can be overcome, but their method is not "it". As said by Bengio, they are partially responsible for the revival of RNNs (alongside Mikolov)

[9]Strategies for Training Large Scale Neural Network Language Models

Tomas Mikolov, A. Deoras, D. Povey, Lukas Burget, Jan Cernocky - 2011

9 papers in library cite

Just builds on other things. Very minor suff in my opinion.

[10]Continuous Space Language Models

Holger Schwenk - 2007

12 papers in library cite

One more paper about speech recog. Nothing special really.

[11]Empirical Evaluation and Combination of Advanced Language Modeling Techniques

Tomas Mikolov, A. Deoras, S. Kombrink, Lukas Burget, Jan Cernocky - 2011

13 papers in library cite

Early work proving that NNs can be good. But very uninteresting overall.

Reference title contains 'et al'

[12]Latent Dirichlet Allocation

D. M. Blei, Andrew Y. Ng, Michael I. Jordan - 2003

10 papers in library cite

30 pages; LDA

[13]Indexing by Latent Semantic Analysis

S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, R. Harshman - 1990

12 papers in library cite

LSA paper

[14]Structured Output Layer neural Network Language Model

H. S. Le, I. Oparin, A. Allauzen, Jean Luc Gauvain, F. Yvon - 2011

7 papers in library cite

[15]Statistical Language Models Based on Neural Networks

Tomas Mikolov - 2012

17 papers in library cite

Mikolov's Thesis

[16]A Cache-Based Natural Language Model for Speech Recognition

R. Kuhn, R. D. Mori - 1990

6 papers in library cite

[17]A Joint Language Model With Fine-Grain Syntactic Tags

D. Filimonov, M. Harper - 2009

4 papers in library cite

[18]Exact Training of a Neural Syntactic Language Model

A. Emami, Frederick Jelinek - 2004

4 papers in library cite

[19]Random Forests and the Data Sparseness Problem in Language Modeling

P. Xu - 2005

4 papers in library cite

[20]The Kaldi Speech Recognition Toolkit

D. Povey, A. Ghoshal - 2011

4 papers in library cite

[21]On the Dynamic Adaptation of Stochastic Language Modeling

R. Kneser, V. Steinbiss - 1993

3 papers in library cite

[22]Self-Supervised Discriminative Training of Statistical Language Models

P. Xu, D. Karakos, Sanjeev Khudanpur - 2009

3 papers in library cite

[23]Shrinking Exponential Language Models

S. F. Chen - 2009

3 papers in library cite

[24]Exploiting Latent Semantic Information in Statistical Language Modeling

J. Bellegarda - 2000

2 papers in library cite

[25]Maximum Entropy Techniques for Exploiting Syntactic, Semantic and Collocational Dependencies in Language Modeling

Sanjeev Khudanpur, Jeffrey Wu - 2000

2 papers in library cite

[26]Modeling Long Distance Dependence in Language: Topic Mixtures Versus Dynamic Cache Models

R. M. Iyer, M. Ostendorf - 1999

2 papers in library cite

[27]Towards Better Integration of Semantic Predictors in Statistical Language Modeling

N. Coccaro, Dan Jurafsky - 1998

2 papers in library cite

[28]Trigger-Based Language Models: A Maximum Entropy Approach

R. Lau, R. Rosenfeld, S. Roukos - 1993

2 papers in library cite

[29]A Whole Sentence Maximum entropy Language Model

R. Rosenfeld - 1997

1 paper in library cites

[30]Cache neural Network Language Models Based on Long-Distance Dependencies for a Spoken Dialog System

F. Z. Martinez, S. E. Boquera, M. J. C. Bleda, R. D. Mori - 2012

1 paper in library cites

[31]Discrimina- Tive Language Modeling With Linguistic and Statistically Derived Features

E. Arisoy, M. Saraclar, B. Roark, I. Shafran - 2012

1 paper in library cites

[32]Improving Arabic Broadcast Transcription Using Automatic Topic Clustering

S. Chu, L. Mangu - 2012

1 paper in library cites

[33]Measuring the Influence of Long Range Dependencies With Neural Network Language Models

L. H. Son, A. Allauzen, F. Yvon - 2012

1 paper in library cites

[34]Performance Prediction for Exponential Language Models

S. Chen - 2009

1 paper in library cites

[35]Personalizing Model M for Voice-Search

Geoffrey Zweig, S. Chang - 2011

1 paper in library cites

[36]Robust Decision Tree State Tying for Continuous Speech Recognition

W. Reichl, W. Chou - 2000

1 paper in library cites

[37]The Subspace Gaussian Mixture Model a Structured Model for Speech Recognition

D. Povey, Lukas Burget - 2011

1 paper in library cites

Cited by

12

papers in your library

Cites

13

papers in your library

Read

on November 23, 2025

Nothing too interesting, just using the context of the RNN.

Tags

Paper Aliases

No aliases