Papperoni

2017

Efficient Softmax Approximation for GPUs

E. Grave, Armand Joulin, M. Cisse, D. Grangier, Hervé Jégou

Open PDF Google Scholar

citations

Cite Score

16

AI summary

This paper introduces an adaptive softmax approach for training language models on GPUs, which forms clusters to minimize computation time and exploits matrix-matrix vector operations, achieving significant efficiency gains and comparable accuracy on EuroParl and One Billion Word datasets.

Main Contributions

Introduces a strategy to produce an approximate hierarchical model that considers the computation time of matrix-matrix multiplications.
Provides an empirical analysis of the model on recent GPUs, leading to a realistic computation time model.
Achieves a significant acceleration factor compared to the regular softmax, with 2x to 10x speed-ups.
Improves accuracy under computational constraints on large corpora.

Abstract

We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computation time. Our approach further reduces the computational time by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax. The code of our method is available at https://github.com/facebookresearch/adaptive-softmax.

Citation Graph

Loading graph...

References [48]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[3]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[4]Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2014

11 papers in library cite

It's a good paper but results are veeeery underwhelming.

[5]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[6]Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

John Duchi, Elad Hazan, Yoram Singer - 2011

19 papers in library cite

I actually skimmed through most of this. It's not a bad paper, but it's a math paper, not AI.

[7]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[8]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[9]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[10]Backpropagation Through Time: What It Does and How to Do It

P. Werbos - 1990

9 papers in library cite

Amazing tutorial! Very pragmatic. Explains very basic concepts and focus on implementation

[11]Improved Backing-Off for M-Gram language Modeling

R. Kneser, Hermann Ney - 1995

11 papers in library cite

It's nice, it's simple... But not NNs and seems very incremental on top of existing backoff

[12]Extensions of Recurrent Neural Network Language Model

Tomas Mikolov, S. Kombrink, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2011

16 papers in library cite

Doesn't add much.

[13]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[14]Hierarchical Probabilistic Neural Network Language Model

F. Morin, Yoshua Bengio - 2005

19 papers in library cite

Nice paper overall. Seems very impactful despite not being as relevant right now.

[15]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[16]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[17]On Using Very Large Target Vocabulary for Neural Machine Translation

Yoshua Bengio - 2014

12 papers in library cite

It's nice, but it starts getting a bit into the realm of "yeah, that seems like a minor improvement". It's nice that they use the importance sampling stuff from the previous paper though - I thought it had completely vanished :)

[18]A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

A. Mnih, Yee Whye Teh - 2012

5 papers in library cite

Too mathy and really did not have a lot of impact TBH

[19]Context Dependent Recurrent Neural Network Language Model

Tomas Mikolov, Geoffrey Zweig - 2012

12 papers in library cite

Nothing too interesting, just using the context of the RNN.

[20]Strategies for Training Large Scale Neural Network Language Models

Tomas Mikolov, A. Deoras, D. Povey, Lukas Burget, Jan Cernocky - 2011

9 papers in library cite

Just builds on other things. Very minor suff in my opinion.

[21]Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, John Makhoul - 2014

9 papers in library cite

Simple and very nice that it showed that NNs can stand on their own for NMT.

[22]Continuous Space Language Models

Holger Schwenk - 2007

12 papers in library cite

One more paper about speech recog. Nothing special really.

[23]Empirical Evaluation and Combination of Advanced Language Modeling Techniques

Tomas Mikolov, A. Deoras, S. Kombrink, Lukas Burget, Jan Cernocky - 2011

13 papers in library cite

Early work proving that NNs can be good. But very uninteresting overall.

[24]Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model

Yoshua Bengio, Jean Sebastien Senecal - 2008

6 papers in library cite

Just a rerun of a previous paper. This is ok, but really not too different.

[25]Quick Training of Probabilistic Neural Nets by Importance Sampling

Yoshua Bengio, Jean Sebastien Senecal - 2003

11 papers in library cite

Good idea to overcome softmax computation cost. Not sure if too relevant today, but definitely better than the 2008 paper that is the same stuff.

[26]Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

Geoffrey E. Hinton, L. Deng, D. Yu, George E. Dahl, A. Mohamed, Navdeep Jaitly, A. Senior, Vincent Vanhoucke, P. Nguyen, T. N. Sainath, Brian Kingsbury - 2012

8 papers in library cite

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

[27]A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton - 2015

2 papers in library cite

[28]Learning Longer Memory in Recurrent Neural Networks

Tomas Mikolov, Armand Joulin, S. Chopra, M. Mathieu, Marc'aurelio Ranzato - 2015

8 papers in library cite

RNNs + longer memory

[29]Structured Output Layer neural Network Language Model

H. S. Le, I. Oparin, A. Allauzen, Jean Luc Gauvain, F. Yvon - 2011

7 papers in library cite

[30]Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation

Holger Schwenk, A. Rousseau, M. Attik - 2012

5 papers in library cite

[31]A Bit of Progress in Language Modeling

J. Goodman - 2001

15 papers in library cite

Focuses on n-grams.

[32]Class-Based N-Gram Models of Natural Language

P. F. Brown, P. V. Desouza, R. L. Mercer, Vincent J. Della Pietra, J. C. Lai - 1992

12 papers in library cite

[33]Estimation of Probabilities From Sparse Data for the Language Model Component of a Speech Recognizer

S. Katz - 1987

11 papers in library cite

[34]Classes for Fast Maximum Entropy Training

J. T. Goodman - 2001

7 papers in library cite

[35]Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models

M. Gutmann, A. Hyvarinen - 2010

7 papers in library cite

[36]A Cache-Based Natural Language Model for Speech Recognition

R. Kuhn, R. D. Mori - 1990

6 papers in library cite

[37]An Efficient Gradient-Based Algorithm for on-Line Training of Recurrent Network Trajectories

Ronald J. Williams, J. Peng - 1990

5 papers in library cite

[38]Decoding With Large-Scale Neural Language Models Improves Translation

Ashish Vaswani, Y. Zhao, V. Fossum, D. Chiang - 2013

5 papers in library cite

[39]A Maximum Likelihood Approach to Continuous Speech Recognition

L. R. Bahl, Frederick Jelinek, R. L. Mercer - 1983

4 papers in library cite

[40]Sparse Non-Negative Matrix Language Modeling for Skip-Grams

Noam Shazeer, J. Pelemans, C. Chelba - 2015

3 papers in library cite

[41]Blackout: Speeding Up Recurrent Neural Network Language Models With Very Large Vocabularies

S. Ji, S. V. N. Vishwanathan, S. Nadathur, M. J. Anderson, P. Dubey - 2015

2 papers in library cite

[42]Efficient Exact Gradient Update for Training Deep Networks With Very Large Sparse Targets

Pascal Vincent, A. D. Brebisson, X. Bouthillier - 2015

2 papers in library cite

[43]Europarl: A Parallel Corpus for Statistical Machine Translation

P. Koehn - 2005

2 papers in library cite

[44]Human Behavior and the Principle of Least Effort

G. K. Zipf - 1949

2 papers in library cite

[45]Speed Regularization and Optimality in Word Classing

Geoffrey Zweig, K. Makarychev - 2013

2 papers in library cite

[46]Strategies for Training Large Vocabulary Neural Language Models

Weizhu Chen, D. Grangier, Michael Auli - 2015

2 papers in library cite

[47]Learning Visual Features From Large Weakly Supervised Data

Armand Joulin, Laurens Van Der Maaten, A. Jabri, N. Vasilache - 2015

1 paper in library cites

[48]When and Why Are Log-Linear Models Self-Normalizing

Jacob Andreas, Dan Klein - 2014

1 paper in library cites

Cited by

4

papers in your library

Cites

28

papers in your library

Read

on November 15, 2025

I loved the methodology and the idea behind it. It's good to see some practical improvements (rather than methodological). I just think it's a bit tough to read. I had to ask AI to help explain some stuff.

Tags

Paper Aliases

No aliases