Papperoni

2007

Continuous Space Language Models

Holger Schwenk

Open PDF Google Scholar

citations

Cite Score

28

AI summary

This paper introduces a neural network language model that estimates probability in continuous space, achieving word error rate reductions on international benchmark tasks using the NIST evaluations on broadcast news and conversational speech recognition. It uses training corpora of several hundred million words.

Main Contributions

Introduces a neural network language model for large vocabulary continuous speech recognition.
Describes highly efficient learning algorithms for training corpora of several hundred million words.
Demonstrates incorporation into a large vocabulary continuous speech recognizer using lattice rescoring.
Achieves consistent word error rate reductions on benchmark tasks.
Compares favorably to four-gram back-off language models with modified Kneser-Ney smoothing.

Abstract

This paper describes the use of a neural network language model for large vocabulary continuous speech recognition. The underlying idea of this approach is to attack the data sparseness problem by performing the language model probability estimation in a continuous space. Highly efficient learning algorithms are described that enable the use of training corpora of several hundred million words. It is also shown that this approach can be incorporated into a large vocabulary continuous speech recognizer using a lattice rescoring framework at a very low additional processing time. The neural network language model was thoroughly evaluated in a state-of-the-art large vocabulary continuous speech recognizer for several international benchmark tasks, in particular the Nist evaluations on broadcast news and conversational speech recognition. The new approach is compared to four-gram back-off language models trained with modified Kneser-Ney smoothing which has often been reported to be the best known smoothing method. Usually the neural network language model is interpolated with the back-off language model. In that way, consistent word error rate reductions for all considered tasks and languages were achieved, ranging from 0.4% to almost 1% absolute.

Citation Graph

Loading graph...

References [60]

Sort:

Filter:

[1]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[2]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[3]Srilm - An Extensible Language Modeling Toolkit

Andreas Stolcke - 2002

13 papers in library cite

Toolkit for N-grams. Not too relevant and sounds veeeery simple (sorry for those who implemented it). It's nice to see early implementation of OOP though. The paper is boring and doesn't really say much about the framework, more of a description of how to use the commands and n-gram models.

[4]A Maximum Entropy Approach to Natural Language Processing

A. L. Berger, S. A. D. Pietra, Vincent J. Della Pietra - 1996

10 papers in library cite

This paper is so good! Easy to follow and very nice results. The experiments are a bit meh, but otherwise wonderful.

[5]Probabilistic Interpretation of Feedforward Classification Network Outputs, With Relationship to Statistical Pattern Recognition

John S. Bridle - 1990

12 papers in library cite

Explains using softmax as output. Nice read, but I am not sure if too relevant.

[6]Learning Distributed Representations of Concepts

Geoffrey E. Hinton - 1986

13 papers in library cite

 Probably seminal, but a bit boring overall

[7]Hierarchical Probabilistic Neural Network Language Model

F. Morin, Yoshua Bengio - 2005

19 papers in library cite

Nice paper overall. Seems very impactful despite not being as relevant right now.

[8]Quick Training of Probabilistic Neural Nets by Importance Sampling

Yoshua Bengio, Jean Sebastien Senecal - 2003

11 papers in library cite

Good idea to overcome softmax computation cost. Not sure if too relevant today, but definitely better than the 2008 paper that is the same stuff.

[9]Connectionist Language Modeling for Large Vocabulary Continuous Speech Recognition

Holger Schwenk, Jean Luc Gauvain - 2002

14 papers in library cite

Only real relevance is being early. Otherwise not much to see.

[10]Training Neural Network Language Models on Very Large Corpora

Holger Schwenk, Jean Luc Gauvain - 2005

7 papers in library cite

Seems very derivative of Schwenk's early work. It's also very focused on speech recognition, and "very large corpora" seems very relative.

[11]Can Artificial Neural Networks Learn Language Models

Weixin Xu, Alex Rudnicky - 2000

5 papers in library cite

Shitty paper and I hate that it was the first.

[12]Continuous Space Language Models for Statistical Machine Translation

Holger Schwenk, D. Dchelotte, Jean Luc Gauvain - 2006

5 papers in library cite

Beginning of translation with NNs. Nothing special but good nonetheless.

[13]Sequential Neural Text Compression

Jürgen Schmidhuber - 1996

3 papers in library cite

I really liked this paper. Maybe not useful, but the idea is very nice!

[14]Efficient Training of Large Neural Networks for Language Modeling

Holger Schwenk - 2004

6 papers in library cite

Very early paper, maybe that's why it's relevant. But very uninteresting despite the nice name ("large NNs" had a very different meaning back then)

[15]Using a Connectionist Model in a Syntactical Based Language Model

Frederick Jelinek - 2003

6 papers in library cite

A copy of the other paper. I still don't get what the "Structured Language Model" is and this seems like trying to fit a cube in a circular hole. I don't like it.

[16]Bagging Predictors

L. Breiman - 1994

4 papers in library cite

20 pages; Bagging, 40k citations

[17]Neural Networks for Pattern Recognition

C. M. Bishop - 1995

12 papers in library cite

Book, 38k citations

[18]Indexing by Latent Semantic Analysis

S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, R. Harshman - 1990

12 papers in library cite

LSA paper

[19]Statistical Learning Theory

V. N. Vapnik - 1998

10 papers in library cite

Book, but there's a 12 page overview

[20]A Bit of Progress in Language Modeling

J. Goodman - 2001

15 papers in library cite

Focuses on n-grams.

[21]An Empirical Study of Smoothing Techniques for Language Modeling

S. F. Chen, J. Goodman - 1998

13 papers in library cite

[22]Class-Based N-Gram Models of Natural Language

P. F. Brown, P. V. Desouza, R. L. Mercer, Vincent J. Della Pietra, J. C. Lai - 1992

12 papers in library cite

[23]Estimation of Probabilities From Sparse Data for the Language Model Component of a Speech Recognizer

S. Katz - 1987

11 papers in library cite

[24]A Cache-Based Natural Language Model for Speech Recognition

R. Kuhn, R. D. Mori - 1990

6 papers in library cite

[25]A Maximum Entropy Approach to Adaptive Statistical Language Modeling

R. Rosenfeld - 1996

6 papers in library cite

[26]Structured Language Modeling

C. Chelba, Frederick Jelinek - 2000

6 papers in library cite

[27]Natural Language Processing With Modular PDP Networks and Distributed Lexicon

R. Miikkulainen, M. G. Dyer - 1991

4 papers in library cite

[28]Random Clusterings for Language Modeling

A. Emami, Frederick Jelinek - 2005

4 papers in library cite

[29]A Statistical Approach to Machine Translation

P. Brown, J. Cocke, S. D. Pietra, Vincent J. Della Pietra, Frederick Jelinek, J. Lafferty, R. Mercer, P. Roossin - 1990

3 papers in library cite

[30]Building Continuous Space Language Models for Transcribing European Languages

Holger Schwenk, Jean Luc Gauvain - 2005

3 papers in library cite

[31]On the Dynamic Adaptation of Stochastic Language Modeling

R. Kneser, V. Steinbiss - 1993

3 papers in library cite

[32]Using Phipac to Speed Error Back-Propagation Learning

J. Bilmes, K. Asanovic, C. Chin, J. Demmel - 1997

3 papers in library cite

[33]A Post-Processing System to Yield Reduced Error Word Rates: Recognizer Output Voting Error Reduction (ROVER)

J. G. Fiscus - 1997

2 papers in library cite

[34]Conversational Telephone Speech Recognition

Jean Luc Gauvain, L. Lamel, Holger Schwenk, G. Adda, L. C. Chen, F. Lefe`vre - 2003

2 papers in library cite

[35]Extracting Distributed Representations of Concepts and Relations From Positive and Negative Propositions

A. Paccanaro, Geoffrey Hinton - 2000

2 papers in library cite

[36]Neural Network Language Models for Conversational Speech Recognition

Holger Schwenk, Jean Luc Gauvain - 2004

2 papers in library cite

[37]Relevance Weighting for Combining Multi-Domain Data for N-Gram Language Modeling

R. Iyer, M. Ostendorf - 1999

2 papers in library cite

[38]The LIMSI Broadcast News Transcription System

Jean Luc Gauvain, L. Lamel, G. Adda - 2002

2 papers in library cite

[39]Using Continuous Space Language Models for Conversational Speech Recognition

Holger Schwenk, Jean Luc Gauvain - 2003

2 papers in library cite

[40]Where Are We in Transcribing BN French?

Jean Luc Gauvain, G. Adda, M. A. Decker, A. Allauzen, V. Gendner, L. Lamel, Holger Schwenk - 2005

2 papers in library cite

[41]A New Metric for Stochastic Language Model Evaluation

A. Ito, M. Khoda, M. Ostendorf - 1999

1 paper in library cites

[42]A Similar Decomposition Is Used in Statistical Machine Translation

P. Brown, J. Cocke, S. D. Pietra, Vincent J. Della Pietra, Frederick Jelinek, J. Lafferty, R. Mercer, P. Roossin - 1990

1 paper in library cites

[43]A Study of English Word Category Prediction Based on Neural Networks

M. Nakamura, K. Shikano - 1989

1 paper in library cites

[44]Bayesian Estimation Methods for N-Gram Language Model Adaptation

M. Federico - 1996

1 paper in library cites

[45]Connectionist N-Gram Models by Using MLPs

M. J. Castro, V. Polvoreda - 2001

1 paper in library cites

[46]Discriminative Training of Language Models for Speech Recognition

H. K. J. Kuoi, E. F. Lussier, H. Jiang, C. H. Lee - 2002

1 paper in library cites

[47]Discriminative Training on Language Model

Ziru Chen, K. F. Lee, M. J. Li - 2000

1 paper in library cites

[48]Getting More Mileage From Web Text Sources for Conversational Speech Language Modeling Using Class-Dependent Mixtures

I. Bulyko, M. Ostendorf, Andreas Stolcke - 2003

1 paper in library cites

[49]Language Modeling in the Continuous Domain

Holger Schwenk - 2001

1 paper in library cites

[50]New Directions in Connectionist Language Modeling

M. J. Castro, F. Prat - 2003

1 paper in library cites

[51]Quick Transcription and Automatic Segmentation of the fisher Conversational Telephone Speech corpus

O. Kimball, C. L. Kao, T. Arvizo, John Makhoul, R. Iyer - 2004

1 paper in library cites

[52]The 2004 20xRT BBN/LIMSI english conversational Telephone Speech System

R. Prasad, S. Matsoukas, C. L. Kao, J. Ma, D. X. Xu, T. Colthurst, G. Thattai, O. Kimball, Richard Schwartz, Jean Luc Gauvain, L. Lamel, Holger Schwenk, G. Adda, F. Lefevre - 2004

1 paper in library cites

[53]The 2004 BBN/LIMSI 10xRT english broadcast news Transcription System

L. Nguyen, S. Abdou, M. Afify, John Makhoul, S. Matsoukas, Richard Schwartz, Bing Xiang, L. Lamel, Jean Luc Gauvain, G. Adda, Holger Schwenk, F. Lefevre - 2004

1 paper in library cites

[54]The Ester Phase II Evaluation Campaign for the Rich Transcription of French Broadcast News

S. Galliano, E. Geoffrois, D. Mostefa, K. Choukri, J. F. Bonastre, G. Gravier - 2005

1 paper in library cites

[55]The LIMSI 2006 TC-STAR Transcription Systems

L. Lamel, Jean Luc Gauvain, G. Adda, C. Barras, E. Bilinski, O. Galibert, A. Pujol, Holger Schwenk, X. Zhu - 2006

1 paper in library cites

[56]The Use of a Linguistically Motivated Language Model in Conversational Speech Recognition

Wenyi Wang, Andreas Stolcke, M. Harper - 2004

1 paper in library cites

[57]Transcribing Lectures and Seminars

L. Lamel, G. Adda, E. Bilinski, Jean Luc Gauvain - 2005

1 paper in library cites

[58]Using Information Retrieval Methods for Language Model Adaptation

L. C. Chen, Jean Luc Gauvain, L. Lamel, G. Adda, M. Adda - 2001

1 paper in library cites

[59]Using Neural Network Language Models for LVCSR

Holger Schwenk, Jean Luc Gauvain - 2004

1 paper in library cites

[60]Using Random Forest Language Models in the IBM RT-04 CTS System

P. Xu, L. Mangu - 2005

1 paper in library cites

Cited by

12

papers in your library

Cites

19

papers in your library

Read

on April 27, 2025

One more paper about speech recog. Nothing special really.

Tags

Paper Aliases

No aliases