Papperoni

2017

Using the Output Embedding to Improve Language Models

O. Press, Lior Wolf

Open PDF Google Scholar

citations

Cite Score

32

AI summary

This paper analyzes the output embedding matrix of neural network language models, revealing it as a valid word embedding. It introduces tying input and output embeddings and a regularization method, achieving reduced perplexity and smaller translation model sizes. It uses LSTM and other neural language models.

Main Contributions

It is shown that the output embedding matrix of neural network language models is a valid word embedding.
Tying the input embedding and the output embedding is recommended when training language models.
A new method of regularizing the output embedding is offered.
Weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance.
The approach leads to a significant reduction in perplexity on a variety of neural network language models.

Abstract

We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying the input embedding and this output embedding. We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model. We also offer a new method of regularizing the output embedding. Our methods lead to a significant reduction in perplexity, as we are able to show on a variety of neural network language models. Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance.

Citation Graph

Loading graph...

References [41]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[5]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[6]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[7]Distilling the Knowledge in a Neural Network

Jeffrey Dean - 2015

6 papers in library cite

Very nice, very relevant.

[8]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[9]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[10]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[11]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[12]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[13]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[14]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[15]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[16]LSTM neural Networks for Language Modeling

M. Sundermeyer, R. Schluter, Hermann Ney - 2010

7 papers in library cite

It reads like an undergrad project - doesn't add anything new.

[17]A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Gal - 2015

9 papers in library cite

The methodology is very simple and effective, but maybe it was too simple for a paper, so they put A TON of math on top of it... Unnecessary really.

[18]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[19]How to Construct Deep Recurrent Neural Networks

Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2013

7 papers in library cite

Very interesting despite not being too relevant. Good read and a new way of thinking about RNNs.

[20]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[21]A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

A. Mnih, Yee Whye Teh - 2012

5 papers in library cite

Too mathy and really did not have a lot of impact TBH

[22]Recurrent Highway Networks

J. G. Zilly, R. K. Srivastava, J. Koutnik, Jürgen Schmidhuber - 2016

6 papers in library cite

Recurrent highway networks - maybe read after the original highway network

[23]Learning Longer Memory in Recurrent Neural Networks

Tomas Mikolov, Armand Joulin, S. Chopra, M. Mathieu, Marc'aurelio Ranzato - 2015

8 papers in library cite

RNNs + longer memory

[24]Improving Neural Networks With Dropout

N. Srivastava - 2013

6 papers in library cite

[25]Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

H. Inan, K. Khosravi, Richard Socher - 2017

6 papers in library cite

[26]Domain Adaptation via Pseudo in-Domain Data Selection

A. Axelrod, X. Fe, Jianfeng Gao - 2011

5 papers in library cite

[27]Edinburgh neural Machine Translation Systems for WMT 16

R. Sennrich, B. Haddow, Alexandra Birch - 2016

5 papers in library cite

[28]Learning Word Embeddings Efficiently With Noise-Contrastive Estimation

A. Mnih, Koray Kavukcuoglu - 2013

4 papers in library cite

[29]A New Algorithm for Data Compression

P. Gage - 1994

3 papers in library cite

[30]Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal, Zoubin Ghahramani - 2015

3 papers in library cite

[31]Simlex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

F. Hill, R. Reichart, Anna Korhonen - 2015

3 papers in library cite

[32]Multimodal Distributional Semantics

E. Bruni, N. K. Tran, M. Baroni - 2014

2 papers in library cite

[33]A Dual Embedding Space Model for Document Ranking

B. Mitra, E. Nalisnick, N. Craswell, Rich Caruana - 2016

1 paper in library cites

[34]A Fixed-Size Encoding Method for Variable-Length Sequences With Its Ap Plication to Neural Network Language Models

S. Zhang, H. Jiang, Mimee Xu, J. Hou, L. R. Dai - 2015

1 paper in library cites

[35]An Unsupervised Model for Instance Level Subcategorization Acquisition

S. Baker, R. Reichart, Anna Korhonen - 2014

1 paper in library cites

[36]Gated Word-Character Recurrent Language Model

Y. Miyamoto, Kyunghyun Cho - 2016

1 paper in library cites

[37]Large-Scale Learning of Word Relatedness With Constraints

G. Halawi, G. Dror, E. Gabrilovich, Y. Koren - 2012

1 paper in library cites

[38]Noisy Activation Functions

C. A. G. Ehre, M. Moczulski, M. Denil, Yoshua Bengio - 2016

1 paper in library cites

[39]Practical Solutions to the Problem of Diagonal Dominance in Kernel Document Clustering

D. Greene, P. Cunningham - 2006

1 paper in library cites

[40]Proceedings of the Seventeenth Conference on Computational Natural Language Learning, Chapter Better Word Representations With Recursive Neural Networks for Morphology

T. Luong, Richard Socher, C. M. Ning - 2013

1 paper in library cites

Reference title contains 'et al'

[41]word2vec Explained: Deriving mikolov et al.’s Negative-Sampling Word-Embedding Method

Y. Goldberg, Omer Levy - 2014

1 paper in library cites

Cited by

7

papers in your library

Cites

23

papers in your library

Read

on August 10, 2025

I did not like this paper at all - The paper is not bad, it's just that I expected *way* more. Good results but uninteresting

Tags

Paper Aliases

No aliases