Papperoni

2016

Improving Neural Language Models With a Continuous Cache

E. Grave, Armand Joulin, Nicolas Usunier

Open PDF Google Scholar

citations

Cite Score

17

AI summary

This paper introduces a neural cache model for neural language models that stores recent hidden activations and accesses them through a dot product. It requires no training and scales effortlessly to thousands of memory cells, demonstrating improved performance on language model tasks and the LAMBADA dataset.

Main Contributions

Proposes a continuous version of the cache model, called Neural Cache Model, that can be adapted to any neural network language model.
Stores recent hidden activations and uses them as representation for the context using simply a dot-product with the current hidden activations.
The model requires no training and can be used on any pre-trained neural networks.
The model scales effortlessly to thousands of memory cells.
Demonstrates the quality of the Neural Cache models on several language model tasks and the LAMBADA dataset.

Abstract

We propose an extension to neural network language models to adapt their prediction to the recent history. Our model is a simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation. This mechanism is very efficient and scales to very large memory sizes. We also draw a link between the use of external memory in neural network and cache models used with count based language models. We demonstrate on several language model datasets that our approach performs significantly better than recent memory augmented networks.

Citation Graph

Loading graph...

References [50]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[3]Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2014

11 papers in library cite

It's a good paper but results are veeeery underwhelming.

[4]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[5]Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

John Duchi, Elad Hazan, Yoram Singer - 2011

19 papers in library cite

I actually skimmed through most of this. It's not a bad paper, but it's a math paper, not AI.

[6]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[7]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[8]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[9]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[10]Backpropagation Through Time: What It Does and How to Do It

P. Werbos - 1990

9 papers in library cite

Amazing tutorial! Very pragmatic. Explains very basic concepts and focus on implementation

[11]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[12]Pointer Networks

Oriol Vinyals, M. Fortunato, Navdeep Jaitly - 2015

10 papers in library cite

Cool concept. Nice that it works and can find good solutions for TSP.

[13]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[14]Pointer Sentinel Mixture Models

S. Merity, Caiming Xiong, J. Bradbury, Richard Socher - 2017

12 papers in library cite

I really liked the methodology, but I had to read it a few times to understand it intuitively - I think they should have done a better job at explaining it.

[15]Neural Turing Machines

Alex Graves, G. Wayne, Ivo Danihelka - 2014

18 papers in library cite

This paper is amazing. If someone told me that NNs could use and address memory by position I wouldn't believe it worked. Very nice, but it's a shame that it's just a toy example.

[16]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[17]Improved Backing-Off for M-Gram language Modeling

R. Kneser, Hermann Ney - 1995

11 papers in library cite

It's nice, it's simple... But not NNs and seems very incremental on top of existing backoff

[18]A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Gal - 2015

9 papers in library cite

The methodology is very simple and effective, but maybe it was too simple for a paper, so they put A TON of math on top of it... Unnecessary really.

[19]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[20]The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez - 2016

12 papers in library cite

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

[21]Context Dependent Recurrent Neural Network Language Model

Tomas Mikolov, Geoffrey Zweig - 2012

12 papers in library cite

Nothing too interesting, just using the context of the RNN.

[22]A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Deli Chen, J. Bolton, Christopher D. Manning - 2016

9 papers in library cite

Very solid work that shows that things are not always what they seem - very nice!

[23]Inferring Algorithmic Patterns With Stack-Augmented Recurrent Nets

Armand Joulin, Tomas Mikolov - 2015

9 papers in library cite

Very underwhelming TBH. I expected more after reading the Neural Turing Machine paper. This reads like "yeah, we lost the race, here's what we were doing before they did something better"

[24]Empirical Evaluation and Combination of Advanced Language Modeling Techniques

Tomas Mikolov, A. Deoras, S. Kombrink, Lukas Burget, Jan Cernocky - 2011

13 papers in library cite

Early work proving that NNs can be good. But very uninteresting overall.

[25]Efficient Softmax Approximation for GPUs

E. Grave, Armand Joulin, M. Cisse, D. Grangier, Hervé Jégou - 2017

4 papers in library cite

I loved the methodology and the idea behind it. It's good to see some practical improvements (rather than methodological). I just think it's a bit tough to read. I had to ask AI to help explain some stuff.

[26]Pointing the Unknown Words

C. G. Gulcehre, S. Ahn, R. Nallapati, B. Zhou, Yoshua Bengio - 2016

7 papers in library cite

Bengio, pointer networks

[27]Recurrent Highway Networks

J. G. Zilly, R. K. Srivastava, J. Koutnik, Jürgen Schmidhuber - 2016

6 papers in library cite

Recurrent highway networks - maybe read after the original highway network

[28]Text Understanding With the Attention Sum Reader Network

R. Kadlec, M. Schmid, O. Bajgar, Jan Kleindienst - 2016

7 papers in library cite

Not too cited but cited a lot internally

[29]Learning Longer Memory in Recurrent Neural Networks

Tomas Mikolov, Armand Joulin, S. Chopra, M. Mathieu, Marc'aurelio Ranzato - 2015

8 papers in library cite

RNNs + longer memory

[30]Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

J. Dodge, A. Gane, X. Zhang, Antoine Bordes, S. Chopra, A. Miller, A. Szlam, Jason Weston - 2015

4 papers in library cite

I like these "prerequisite" papers

[31]A Bit of Progress in Language Modeling

J. Goodman - 2001

15 papers in library cite

Focuses on n-grams.

[32]Estimation of Probabilities From Sparse Data for the Language Model Component of a Speech Recognizer

S. Katz - 1987

11 papers in library cite

[33]The Mathematics of Statistical Machine Translation: Parameter Estimation

P. F. Brown, S. D. Pietra, Vincent J. Della Pietra, R. L. Mercer - 1993

7 papers in library cite

[34]A Cache-Based Natural Language Model for Speech Recognition

R. Kuhn, R. D. Mori - 1990

6 papers in library cite

[35]A Maximum Entropy Approach to Adaptive Statistical Language Modeling

R. Rosenfeld - 1996

6 papers in library cite

[36]An Efficient Gradient-Based Algorithm for on-Line Training of Recurrent Network Trajectories

Ronald J. Williams, J. Peng - 1990

5 papers in library cite

[37]Learning to Transduce With Unbounded Memory

Edward Grefenstette, K. Hermann, M. Suleyman, Phil Blunsom - 2015

5 papers in library cite

[38]A Maximum Likelihood Approach to Continuous Speech Recognition

L. R. Bahl, Frederick Jelinek, R. L. Mercer - 1983

4 papers in library cite

[39]Larger-Context Language Modelling

Tianle Wang, Kyunghyun Cho - 2015

4 papers in library cite

[40]A Dynamic Language Model for Speech Recognition

Frederick Jelinek, B. Merialdo, S. Roukos, M. Strauss - 1991

3 papers in library cite

[41]On the Dynamic Adaptation of Stochastic Language Modeling

R. Kneser, V. Steinbiss - 1993

3 papers in library cite

[42]Exploiting Latent Semantic Information in Statistical Language Modeling

J. Bellegarda - 2000

2 papers in library cite

[43]Maximum Entropy Techniques for Exploiting Syntactic, Semantic and Collocational Dependencies in Language Modeling

Sanjeev Khudanpur, Jeffrey Wu - 2000

2 papers in library cite

[44]Modeling Long Distance Dependence in Language: Topic Mixtures Versus Dynamic Cache Models

R. M. Iyer, M. Ostendorf - 1999

2 papers in library cite

[45]Towards Better Integration of Semantic Predictors in Statistical Language Modeling

N. Coccaro, Dan Jurafsky - 1998

2 papers in library cite

[46]Trigger-Based Language Models: A Maximum Entropy Approach

R. Lau, R. Rosenfeld, S. Roukos - 1993

2 papers in library cite

[47]Adaptive Language Modeling Using Minimum Discriminant Estimation

S. D. Pietra, V. D. Pietra, R. L. Mercer, S. Roukos - 1992

1 paper in library cites

[48]Dialogue act Modeling for Automatic Tagging and Recognition of Conversational Speech

Andreas Stolcke, N. Coccaro, R. Bates, P. Taylor, C. V. E. Dykema, K. Ries, E. Shriberg, Dan Jurafsky, R. Martin, M. Meteer - 2000

1 paper in library cites

[49]Probabilistic Models of Short and Long Distance Word Dependencies in Running Text

J. Kupiec - 1989

1 paper in library cites

[50]Speech recognition and the Frequency of Recently Used Words: A Modified Markov Model for Natural Language

R. Kuhn - 1988

1 paper in library cites

Cited by

7

papers in your library

Cites

30

papers in your library

Read

on October 31, 2025

This was a surprise to me - I expected this to suck. However, they can provide a simple and intuitive way of improving LMs - nice

Tags

Paper Aliases

No aliases