Papperoni

2012

Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

G. Dahl, D. Yu, L. Deng, Alex Acero

Open PDF Google Scholar

citations

Cite Score

70

AI summary

This paper introduces a novel context-dependent deep neural network hidden Markov model (CD-DNN-HMM) for large-vocabulary speech recognition (LVSR), leveraging deep belief networks for phone recognition. The model achieves significant improvements on a business search dataset, outperforming conventional GMM-HMMs.

Main Contributions

Proposes a novel CD-DNN-HMM hybrid architecture for LVSR.
Demonstrates that CD-DNN-HMMs can significantly outperform conventional CD-GMM-HMMs on a challenging business search dataset.
Achieves an absolute sentence accuracy improvement of 5.8% and 9.2% over CD-GMM-HMMs.
Illustrates the key components of the CD-DNN-HMM model and describes the procedure for applying it to LVSR.
Analyzes the effects of various modeling choices on performance.

Abstract

We propose a novel context-dependent (CD) model for large-vocabulary speech recognition (LVSR) that leverages recent advances in using deep belief networks for phone recognition. We describe a pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output. The deep belief network pre-training algorithm is a robust and often helpful way to initialize deep neural networks generatively that can aid in optimization and reduce generalization error. We illustrate the key components of our model, describe the procedure for applying CD-DNN-HMMs to LVSR, and analyze the effects of various modeling choices on performance. Experiments on a challenging business search dataset demonstrate that CD-DNN-HMMs can significantly outperform the conventional context-dependent Gaussian mixture model (GMM)-HMMs, with an absolute sentence accuracy improvement of 5.8% and 9.2% (or relative error reduction of 16.0% and 23.2%) over the CD-GMM-HMMs trained using the minimum phone error rate (MPE) and maximum-likelihood (ML) criteria, respectively.

Citation Graph

Loading graph...

References [84]

Sort:

Filter:

[1]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[2]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[3]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[4]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[5]Learning Deep Architectures for AI

Yoshua Bengio - 2009

25 papers in library cite

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

[6]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[7]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[8]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Good read, but I think I need to revisit it after I understand RBMs better.

[9]Why Does Unsupervised Pre-Training Help Deep Learning?

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre Antoine Manzagol, Pascal Vincent, Samy Bengio - 2010

12 papers in library cite

Good paper, easy to follow, and brings some light to the pre-training stuff (layer-by-layer). I just wish it wasn't so long. It's a chore.

[10]What Is the Best Multi-Stage Architecture for Object Recognition?

K. Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun - 2009

20 papers in library cite

So boring and convoluted... They try to make things more abstract by saying "filter banks" and "2 stages", but in the end it's CNNs. It seems like they are trying to tie NNs to the rest of ML, but it doesn't work.

[11]Scaling Learning Algorithms Towards AI

Yoshua Bengio, Yann Lecun - 2007

15 papers in library cite

I should have read this sooner! Such a good explanation of why deep learning > other stuff! Also, better than Bengio's 2006 Learning Deep Archs for AI

[12]Deep Learning via Hessian-Free Optimization

James Martens - 2010

12 papers in library cite

This paper is surprisingly good! When I first read the Hessian-Free optimization part, I thought "ugh, this is going to be full of math", but in the end it was very very enjoyable. I think I just wouldn't give it a 5 because it doesn't seem to have had that much impact.

[13]The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training

Pascal Vincent - 2009

5 papers in library cite

Very nice analysis of why supervised pretraining works!

[14]Information Processing in Dynamical Systems: Foundations of Harmony Theory

P. Smolensky - 1986

11 papers in library cite

88 pages; Introduced RBMs

[15]Deep Belief Networks for Phone Recognition

A. Mohamed, G. Dahl, Geoffrey Hinton - 2009

3 papers in library cite

Cited by hinton in the paper w/ google, ibm, microsoft

[16]Acoustic Modeling Using Deep Belief Networks

A. Mohamed, G. Dahl, Geoffrey Hinton - 2012

12 papers in library cite

[17]Connectionist Speech Recognition: A Hybrid Approach

H. Bourlard, N. Morgan - 1993

8 papers in library cite

[18]Exponential Family Harmoniums With an Application to Information Retrieval

M. Welling, M. R. Zvi, Geoffrey Hinton - 2005

8 papers in library cite

[19]The Curse of Highly Variable Functions for Local Kernel Machines

Yoshua Bengio, O. Delalleau, N. L. Roux - 2006

7 papers in library cite

[20]Phone Recognition With the Mean-Covariance Restricted Boltzmann Machine

George E. Dahl, Marc'aurelio Ranzato, A. Mohamed, Geoffrey E. Hinton - 2010

6 papers in library cite

[21]CUDAMat: A CUDA-based Matrix Class for Python

V. Mnih - 2009

5 papers in library cite

[22]Semantic Hashing

Ruslan Salakhutdinov, Geoffrey Hinton - 2007

5 papers in library cite

[23]The Recurrent Temporal Restricted Boltzmann Machine

Ilya Sutskever, Geoffrey Hinton, G. Taylor - 2008

5 papers in library cite

[24]Boosted Mmi for Model and Feature-Space Discriminative Training

D. Povey, D. Kanevsky, Brian Kingsbury, Bhuvana Ramabhadran, G. Saon, K. Visweswariah - 2008

4 papers in library cite

[25]Continuous Speech Recognition Using Multilayer Perceptrons With Hidden Markov Models

N. Morgan, H. Bourlard - 1990

3 papers in library cite

[26]Decision Trees Do Not Generalize to New Variations

Yoshua Bengio, O. Delalleau, C. Simard - 2007

3 papers in library cite

[27]Investigation of Full-Sequence Training of Deep Belief Networks for Speech Recognition

A. Mohamed, D. Yu, L. Deng - 2010

3 papers in library cite

[28]Roles of Pre-Training and Fine-Tuning in Context-Dependent DBN-HMMS for Real-World Speech Recognition

D. Yu, L. Deng, G. Dahl - 2010

3 papers in library cite

[29]3-D Object Recognition With Deep Belief Nets

V. Nair, Geoffrey Hinton - 2009

2 papers in library cite

[30]CDNN: A Context-Dependent Neural Network for Continuous Speech Recognition

H. Bourlard, N. Morgan, C. Wooters, S. Renals - 1992

2 papers in library cite

[31]Computational Models for Speech Production

L. Deng - 1999

2 papers in library cite

[32]Connectionist Probability Estimators in HMM Speech Recognition

S. Renals, N. Morgan, H. Boulard, Michael Cohen, H. Franco - 1994

2 papers in library cite

[33]Discriminative Learning in Sequential Pattern Recognition — A Unifying Review for Optimization-Oriented Speech Recognition

X. He, L. Deng, W. Chou - 2008

2 papers in library cite

[34]Minimum Phone Error and I-Smoothing for Improved Discriminative Training

D. Povey, P. Woodland - 2002

2 papers in library cite

[35]MMI Training for Continuous Phoneme Recognition on the TIMIT Database

S. Kapadia, V. Valtchev, S. Young - 1993

2 papers in library cite

[36]Modeling Pixel Means and Covariances Using Factorized Third-Order Boltzmann machines

Marc'aurelio Ranzato, Geoffrey Hinton - 2010

2 papers in library cite

[37]Research Developments and Directions in Speech Recognition and Understanding, Part 1

J. Baker, L. Deng, J. Glass, Sanjeev Khudanpur, C. Lee, N. Morgan, D. O'shaugnessy - 2009

2 papers in library cite

[38]Research Developments and Directions in Speech Recognition and Understanding, Part 1

J. Baker, L. Deng, J. Glass, Sanjeev Khudanpur, C. Lee, N. Morgan, D. O'shaugnessy - 2009

2 papers in library cite

[39]Speech Recognition Using Augmented Conditional Random Fields

Y. Hifny, S. Renals - 2009

2 papers in library cite

[40]Structured Speech Modeling

L. Deng, D. Yu, Alex Acero - 2006

2 papers in library cite

[41]Tandem Connectionist Feature Extraction for Conventional HMM Systems

H. Hermansky, D. P. W. Ellis, S. S. Sharma - 2000

2 papers in library cite

[42]A Bidirectional Target Filtering Model of Speech Coarticulation: Two-Stage Implementation for Phonetic Recognition

L. Deng, D. Yu, Alex Acero - 2006

1 paper in library cites

[43]A Comparative Large Scale Study of MLP Features for mandarin ASR

F. Valente, M. Doss, C. Plahl, S. Ravuri, Wenyi Wang - 2010

1 paper in library cites

[44]A Dynamic, Feature-Based Approach to the Interface Between Phonology and Phonetics for Speech Modeling and Recognition

L. Deng - 1998

1 paper in library cites

[45]A Log-Linear Discriminative Modeling Framework for Speech Recognition

Georg Heigold - 2010

1 paper in library cites

[46]A Novel Framework and Training algorithm for Variable-Parameter Hidden Markov Models

D. Yu, L. Deng, Y. Gong, Alex Acero - 2009

1 paper in library cites

[47]A Segmental Conditional Random Field Toolkit for Speech Recognition

Geoffrey Zweig, P. Nguyen - 2010

1 paper in library cites

[48]A Segmental CRF Approach to Large Vocabulary Continuous Speech Recognition

Geoffrey Zweig, P. Nguyen - 2009

1 paper in library cites

[49]A Survey of hybrid ANN/HMM Models for Automatic Speech Recognition

E. Trentin, M. Gori - 2001

1 paper in library cites

[50]A Unified Framework of HMM Adaptation With Joint Compensation of Additive and Convolutive Distortions

Jeffrey Li, L. Deng, D. Yu, Y. Gong, Alex Acero - 2009

1 paper in library cites

[51]An Investigation of Segmental Hidden Dynamic Models of Speech Coarticulation for Automatic Speech Recognition

J. Bridle, L. Deng, J. Picone, H. Richards, J. Ma, T. Kamm, M. Schuster, S. Pike, R. Regan - 1998

1 paper in library cites

[52]Automated Directory Assistance System—From Theory to Practice

D. Yu, Y. Ju, Yuzhi Wang, Geoffrey Zweig, Alex Acero - 2007

1 paper in library cites

[53]Combining Phonetic Attributes Using Conditional Random Fields

J. Morris, E. F. Lussier - 2006

1 paper in library cites

[54]Connectionist Speech Recognition of Broadcast News

A. Robinson, G. Cook, D. Ellis, E. F. Lussier, S. Renals, D. Williams - 2002

1 paper in library cites

[55]Context-Dependent Connectionist Probability Estimation in a Hybrid Hidden Markov model-neural Net Speech Recognition System

H. Franco, Michael Cohen, N. Morgan, D. Rumelhart, V. Abrash - 1994

1 paper in library cites

[56]Continuous Speech Recognition by Connectionist Statistical Methods

H. Boulard, N. Morgan - 1993

1 paper in library cites

[57]Deep-Structured Hidden Conditional Random Fields for Phonetic Recognition

D. Yu, L. Deng - 2010

1 paper in library cites

[58]Development of the SRI/Nightingale Arabic ASR system

D. Vergyri, A. Mandal, Wenyi Wang, Andreas Stolcke, James Zheng, M. Graciarena, D. Rybach, C. Gollan, R. Schluter, K. Kirchhoff, A. Faria, N. Morgan - 2008

1 paper in library cites

[59]Discriminative Training for Large Vocabulary Speech Recognition

D. Povey - 2003

1 paper in library cites

[60]Discriminative Training for Large Vocabulary Speech Recognition Using Minimum Classification Error

E. Mcdermott, T. Hazen, J. Roux, A. Nakamura, S. Katagiri - 2007

1 paper in library cites

[61]Estimation of Global Posteriors and Forward-Backward Training of Hybrid HMM/ANN systems

J. Hennebert, C. Ris, H. Bourlard, S. Renals, N. Morgan - 1997

1 paper in library cites

[62]Hidden Conditional Random Fields for Phone Classification

A. Gunawardana, M. Mahajan, Alex Acero, J. Platt - 2005

1 paper in library cites

[63]Incorporating Training Errors for Large Margin HMMs Under Semi-Definite Programming Framework

H. Jiang, Xiang Lisa Li - 2007

1 paper in library cites

[64]Large Margin Gaussian Mixture Modeling for Phonetic Classification and Recognition

F. Sha, L. Saul - 2006

1 paper in library cites

[65]Large Margin HMMs for Speech Recognition

Xiang Lisa Li, H. Jiang, C. L. Liu - 2005

1 paper in library cites

[66]Large Vocabulary Continuous Speech Recognition With Context-Dependent DBN-HMMs

G. Dahl, D. Yu, L. Deng, Alex Acero - 2011

1 paper in library cites

[67]Large-Margin Discriminative Training of Hidden Markov Models for Speech Recognition

D. Yu, L. Deng - 2007

1 paper in library cites

[68]Large-Margin Minimum Classification Error Training a Theoretical Risk Minimization Perspective

D. Yu, L. Deng, X. He, Alex Acero - 2008

1 paper in library cites

[69]Large-Margin Minimum Classification Error Training for Large-Scale Speech Recognition Tasks

D. Yu, L. Deng, X. He, Alex Acero - 2007

1 paper in library cites

[70]Learning to Detect Roads in High-Resolution Aerial Images

V. Mnih, Geoffrey Hinton - 2010

1 paper in library cites

[71]Live Search for Mobile: Web Services by Voice on the Cellphone

Alex Acero, N. Bernstein, R. Chambers, Y. Ju, Xiang Lisa Li, J. Odell, P. Nguyen, O. Scholtz, Geoffrey Zweig - 2008

1 paper in library cites

[72]Mean and Variance Adaptation Within the MLLR Framework

M. Gales, P. Woodland - 1996

1 paper in library cites

[73]Minimum Classification Error Rate Methods for Speech Recognition

B. Juang, W. Chou, C. Lee - 1997

1 paper in library cites

[74]On Adaptive Decision Rules and Decision Parameter Adaptation for Automatic Speech Recognition

C. Lee, Q. Huo - 2000

1 paper in library cites

[75]Optimizing Bottle-Neck Features for LVCSR

F. Grezl, P. Fousek - 2008

1 paper in library cites

[76]Probabilistic and Bottle-Neck Features for LVCSR of Meetings

F. Grezl, M. Karafiat, S. Kontar, Jan Cernocky - 2007

1 paper in library cites

[77]Pushing the Envelope-Aside

N. Morgan, Qihao Zhu, Andreas Stolcke, K. Sonmez, S. Sivadas, T. Shinozaki, M. Ostendorf, P. Jain, H. Hermansky, D. Ellis, G. Doddington, Berlin Chen, O. Cetin, H. Bourlard, M. Athineos - 2005

1 paper in library cites

[78]Shared-Distribution Hidden Markov Models for Speech Recognition

M. Hwang, X. Huang - 1993

1 paper in library cites

[79]Speaker Adaptation for Hybrid HMM-ANN continuous Speech Recognition System

J. Neto, L. Almeida, M. Hochberg, C. Martins, L. Nunes, S. Renals, Tony Robinson - 1995

1 paper in library cites

[80]Speaker-Adaptation in a Hybrid HMM-MLP Recognizer

J. Neto, C. Martins, L. Almeida - 1996

1 paper in library cites

[81]Speech Recognition Using Neural Networks With Forward-Backward Probability Generated Targets

Y. Yan, M. Fanty, R. Cole - 1997

1 paper in library cites

[82]Transcribing Broadcast Data Using MLP Features

P. Fousek, L. Lamel, J. Gauvain - 2008

1 paper in library cites

[83]Use of Incrementally Regulated Discriminative Margins in MCE Training for Speech Recognition

D. Yu, L. Deng, X. He, Alex Acero - 2006

1 paper in library cites

[84]Using MLP Features in SRI's Conversational Speech Recognition System

Qihao Zhu, Andreas Stolcke, Berlin Chen, N. Morgan - 2005

1 paper in library cites

Cited by

19

papers in your library

Cites

16

papers in your library

Read

on July 11, 2025

Good paper, very well written and probably the best explanation of RBMs and DBNs I've seen. However, I don't see a lot of impact and seems very derivative from other works.

Tags

Paper Aliases

No aliases