Papperoni

2012

Deep Neural Networks for Acoustic Modeling in Speech Recognition

Geoffrey Hinton

Open PDF Google Scholar

citations

Cite Score

88

AI summary

This paper reviews the use of Deep Neural Networks (DNNs) for acoustic modeling in speech recognition, demonstrating that DNNs outperform Gaussian Mixture Models (GMMs) on TIMIT and large vocabulary tasks, using new methods to train DNNs with multiple hidden layers.

Main Contributions

Introduces a two-stage training procedure for fitting DNNs: generative pre-training and discriminative fine-tuning.
Demonstrates that DNNs can outperform GMMs at acoustic modeling for speech recognition on a variety of datasets.
Shows that pre-training the DBN-DNN leads to the best results, but it is not critical, especially when using five or more hidden layers.
Details experiments on TIMIT database, Bing-Voice-Search, Switchboard, Youtube, and English-Broadcast-News.
Discusses alternative pre-training and fine-tuning methods for DNNs.

Abstract

Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input. An alternative way to evaluate the fit is to use a feed-forward neural network that takes several frames of coefficients as input and produces posterior probabilities over HMM states as output. Deep neural networks with many hidden layers, that are trained using new methods have been shown to outperform Gaussian mixture models on a variety of speech recognition benchmarks, sometimes by a large margin. This paper provides an overview of this progress and represents the shared views of four research groups who have had recent successes in using deep neural networks for acoustic modeling in speech recognition.

Citation Graph

Loading graph...

References [68]

Sort:

Filter:

[1]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[2]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[3]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[4]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[5]Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network With a Local Denoising Criterion

P. H. Vincent, Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre Antoine Manzagol - 2010

6 papers in library cite

This is basically a summary of everything that happened from 2006-2010, and also points some interesting things about DBNs! Very well explained as well.

[6]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Good read, but I think I need to revisit it after I understand RBMs better.

[7]Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

G. Dahl, D. Yu, L. Deng, Alex Acero - 2012

19 papers in library cite

Good paper, very well written and probably the best explanation of RBMs and DBNs I've seen. However, I don't see a lot of impact and seems very derivative from other works.

[8]An Empirical Evaluation of Deep Architectures on Problems With Many Factors of Variation

Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio - 2007

13 papers in library cite

Good paper showing promising results for Deep Learning. Nothing amazing but good nonetheless

[9]Deep Learning via Hessian-Free Optimization

James Martens - 2010

12 papers in library cite

This paper is surprisingly good! When I first read the Hessian-Free optimization part, I thought "ugh, this is going to be full of math", but in the end it was very very enjoyable. I think I just wouldn't give it a 5 because it doesn't seem to have had that much impact.

[10]Improving the Speed of Neural Networks on Cpus

Vincent Vanhoucke, A. Senior, Mark Z. Mao - 2011

4 papers in library cite

It's good and it's interesting, but I don't think it adds a ton. Good read though. I read it because I thought it discussed quantization (which it does, but not in the sense of making NNs smaller)

[11]An Application to Recurrent Nets to Phone Probability Estimation

A. Robinson - 1994

9 papers in library cite

Very early work. Seems like a good overview of how to apply NNs to speech recog, but TBH there's nothing impressive here.

[12]Application of Pretrained Deep Neural Networks to Large Vocabulary Speech Recognition

Navdeep Jaitly, P. Nguyen, A. Senior, Vincent Vanhoucke - 2012

6 papers in library cite

It's not bad, it's just nothing new really. They just get existing methods and apply to very large datasets. I see the contribution, but boring read - just experiment methodology and results.

[13]Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition

Dan C. Ciresan, Ueli Meier, Luca M. Gambardella, Jürgen Schmidhuber - 2010

10 papers in library cite

It's short, simple and straight to the point (as the network in the paper). It's refreshing to read something less "academic" and more "in your face"

[14]A Practical Guide to Training Restricted boltzmann Machines

Geoffrey E. Hinton - 2010

4 papers in library cite

[15]Deep Belief Networks for Phone Recognition

A. Mohamed, G. Dahl, Geoffrey Hinton - 2009

3 papers in library cite

Cited by hinton in the paper w/ google, ibm, microsoft

[16]Scalable Stacking and Learning for Building Deep Architectures

L. Deng, D. Yu, J. Platt - 2012

2 papers in library cite

DSN? Sounds interesting

[17]Acoustic Modeling Using Deep Belief Networks

A. Mohamed, G. Dahl, Geoffrey Hinton - 2012

12 papers in library cite

[18]Connectionist Speech Recognition: A Hybrid Approach

H. Bourlard, N. Morgan - 1993

8 papers in library cite

[19]Phone Recognition With the Mean-Covariance Restricted Boltzmann Machine

George E. Dahl, Marc'aurelio Ranzato, A. Mohamed, Geoffrey E. Hinton - 2010

6 papers in library cite

[20]Boosted Mmi for Model and Feature-Space Discriminative Training

D. Povey, D. Kanevsky, Brian Kingsbury, Bhuvana Ramabhadran, G. Saon, K. Visweswariah - 2008

4 papers in library cite

[21]Contracting Auto-Encoders: Explicit Invariance During Feature Extraction

S. Rifai, Pascal Vincent, X. Muller, Xavier Glorot, Yoshua Bengio - 2011

4 papers in library cite

[22]Conversational Speech Transcription Using Context-Dependent Deep Neural Networks

F. Seide, G. Li, D. Yu - 2011

4 papers in library cite

[23]Maximum Mutual Information Estimation of Hidden Markov Model Parameters for Speech Recognition

L. Bahl, P. Brown, P. D. Souza, R. Mercer - 1986

4 papers in library cite

[24]On Optimization Methods for Deep Learning

Quoc V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, Andrew Y. Ng - 2011

4 papers in library cite

[25]Applying Convolutional Neural Networks Concepts to Hybrid NN-HMM Model for Speech Recognition

O. A. Hamid, A. Mohamed, H. Jiang, G. Penn - 2012

3 papers in library cite

[26]Auto-Encoder Bottleneck Features Using Deep Belief Networks

T. N. Sainath, Brian Kingsbury, Bhuvana Ramabhadran - 2012

3 papers in library cite

[27]Investigation of Full-Sequence Training of Deep Belief Networks for Speech Recognition

A. Mohamed, D. Yu, L. Deng - 2010

3 papers in library cite

[28]Lattice-Based Optimization of Sequence Classification Criteria for Neural-Network Acoustic Modeling

Brian Kingsbury - 2009

3 papers in library cite

[29]Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference

J. Pearl - 1988

3 papers in library cite

[30]Roles of Pre-Training and Fine-Tuning in Context-Dependent DBN-HMMS for Real-World Speech Recognition

D. Yu, L. Deng, G. Dahl - 2010

3 papers in library cite

[31]Speech Recognition With Segmental Conditional Random Fields: A Summary of the JHU CLSP 2010 Summer Workshop

Geoffrey Zweig, P. Nguyen, D. V. Compernolle, K. Demuynck, L. Atlas, Peter Clark, G. Sell, Mingliang Wang, F. Sha, H. Hermansky, D. Karakos, A. Jansen, S. Thomas, G. S. V. S. Sivaram, S. Bowman, J. Kao - 2011

3 papers in library cite

[32]Computational Models for Speech Production

L. Deng - 1999

2 papers in library cite

[33]Deep Belief Networks Using Discriminative Features for Phone Recognition

A. Mohamed, T. N. Sainath, George E. Dahl, Bhuvana Ramabhadran, Geoffrey E. Hinton, M. Picheny - 2011

2 papers in library cite

[34]Deep Convex Network: A Scalable Architecture for Speech Pattern Classification

L. Deng, D. Yu - 2011

2 papers in library cite

[35]Discriminative Learning in Sequential Pattern Recognition — A Unifying Review for Optimization-Oriented Speech Recognition

X. He, L. Deng, W. Chou - 2008

2 papers in library cite

[36]Feature Engineering in Context-Dependent Deep Neural Networks for Conversational Speech Transcription

F. Seide, G. Li, X. Chen, D. Yu - 2011

2 papers in library cite

[37]Global Optimization of a Neural Network - Hidden Markov Model Hybrid

Yoshua Bengio, R. D. Mori, G. Flammia, F. Kompe - 1991

2 papers in library cite

[38]Heterogeneous Acoustic Measurements and Multiple Classifiers for Speech Recognition

A. K. Halberstadt - 1998

2 papers in library cite

[39]Improved Pre-Training of Deep Belief Networks Using Sparse Encoding Symmetric Machines

C. Plahl, T. N. Sainath, Bhuvana Ramabhadran, D. Nahamoo - 2012

2 papers in library cite

[40]Making Deep Belief Networks Effective for Large Vocabulary Continuous Speech Recognition

T. N. Sainath, Brian Kingsbury, Bhuvana Ramabhadran, P. Fousek, P. Novak, A. Mohamed - 2011

2 papers in library cite

[41]Research Developments and Directions in Speech Recognition and Understanding, Part 1

J. Baker, L. Deng, J. Glass, Sanjeev Khudanpur, C. Lee, N. Morgan, D. O'shaugnessy - 2009

2 papers in library cite

[42]Speech Recognition Using Augmented Conditional Random Fields

Y. Hifny, S. Renals - 2009

2 papers in library cite

[43]Structured Speech Modeling

L. Deng, D. Yu, Alex Acero - 2006

2 papers in library cite

[44]Tandem Connectionist Feature Extraction for Conventional HMM Systems

H. Hermansky, D. P. W. Ellis, S. S. Sharma - 2000

2 papers in library cite

[45]Understanding How Deep Belief Networks Perform Acoustic Modelling

A. Mohamed, Geoffrey Hinton, G. Penn - 2012

2 papers in library cite

[46]Unsupervised Feature Learning for Audio Classification Using Convolutional Deep Belief Networks

Honglak Lee, P. Pham, Y. Largman, A. Ng - 2009

2 papers in library cite

[47]A Deep Architecture With Bilinear Modeling of Hidden Representations: Applications to Phonetic Recognition

B. Hutchinson, L. Deng, D. Yu - 2012

1 paper in library cites

[48]A Statistical Approach to Automatic Speech Recognition Using the Atomic Speech Units Constructed From Overlapping Articulatory Features

L. Deng, D. Sun - 1994

1 paper in library cites

[49]An Exploration of Large Vocabulary Tools for Small Vocabulary Phonetic Recognition

T. N. Sainath, Bhuvana Ramabhadran, M. Picheny - 2009

1 paper in library cites

[50]An Overlapping-Feature Based Phonological Model Incorporating Linguistic Constraints: Applications to Speech Recognition

Jian Sun, L. Deng - 2002

1 paper in library cites

[51]Backpropagation Training for multilayer conditional Random Field Based Phone Recognition

R. Prabhavalkar, E. F. Lussier - 2010

1 paper in library cites

[52]Boosting Attribute and Phone Estimation Accuracies With Deep Neural Networks for Detection-Based Speech Recognition

D. Yu, S. Siniscalchi, L. Deng, C. Lee - 2012

1 paper in library cites

[53]Cepstral Analysis Technique for Automatic Speaker Verification

S. Furui - 1981

1 paper in library cites

[54]Comparing Multilayer Perceptron to Deep Belief Network Tandem Features for Robust Asr

Oriol Vinyals, S. V. Ravuri - 2011

1 paper in library cites

[55]Deep and Wide: Multiple Layers in Automatic Speech Recognition

N. Morgan - 2012

1 paper in library cites

[56]Digital Speech Processing, Synthesis, and Recognition

S. Furui - 2000

1 paper in library cites

[57]Discriminative Pre-Training of Deep Neural Networks

D. Yu, L. Deng, G. Li, F. Seide - 2011

1 paper in library cites

[58]Exemplar-Based Sparse Representation Features: From Timit to LVCSR

T. N. Sainath, Bhuvana Ramabhadran, M. Picheny, D. Nahamoo, D. Kanevsky - 2011

1 paper in library cites

[59]Improved Phone Recognition Using Bayesian Triphone Models

J. Ming, F. J. Smith - 1998

1 paper in library cites

[60]Large Scale Discriminative Training of Hidden Markov Models for Speech Recognition

P. C. Woodland, D. Povey - 2002

1 paper in library cites

[61]Large Vocabulary Continuous Speech Recognition: A Review

S. Young - 1996

1 paper in library cites

[62]Maximum Likelihood Estimation for Multivariate Mixture Observations of Markov Chains

B. H. Juang, S. Levinson, M. Sondhi - 1986

1 paper in library cites

[63]Perceptual Linear Predictive (plp) Analysis of Speech

H. Hermansky - 1990

1 paper in library cites

[64]Product of Experts for Statistical Parametric Speech Synthesis

H. Zen, M. Gales, Y. Nankaku, K. Tokuda - 2012

1 paper in library cites

[65]Pushing the Envelope Aside [Speech Recognition]

N. Morgan, Qihao Zhu, Andreas Stolcke, K. Sonmez, S. Sivadas, T. Shinozaki, M. Ostendorf, P. Jain, H. Hermansky, D. Ellis, G. Doddington, Berlin Chen, O. Cretin, H. Bourlard, M. Athineos - 2005

1 paper in library cites

[66]Sparse multilayer perceptron for Phoneme Recognition

G. S. V. S. Sivaram, H. Hermansky - 2012

1 paper in library cites

[67]Switching Dynamic System Models for Speech Articulation and Acoustics

L. Deng - 2003

1 paper in library cites

[68]Use of Differential Cepstra as Acoustic Features in Hidden Trajectory Modelling for Phonetic Recognition

L. Deng, D. Yu - 2007

1 paper in library cites

Cited by

21

papers in your library

Cites

17

papers in your library

Read

on July 18, 2025

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

Tags

Paper Aliases

Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups