Papperoni

2016

Can Active Memory Replace Attention?

Lukasz Kaiser, Samy Bengio

Open PDF Google Scholar

citations

Cite Score

5

AI summary

This paper introduces an extension of the Neural GPU model with active memory, achieving good results for neural machine translation. The model clarifies the relationship between attention and active memory and demonstrates its effectiveness, particularly on longer sentences, outperforming traditional attention mechanisms.

Main Contributions

The paper introduces an extension of the Neural GPU model for neural machine translation.
It provides insights into the relationship between attention mechanisms and active memory models.
The model demonstrates competitive performance in machine translation tasks compared to attention models.
It shows that active memory models are less sensitive to sentence length than attention-based models.
The paper highlights the importance of recurrent dependence in generating output distributions for improved performance.

Abstract

Several mechanisms to focus attention of a neural network on selected parts of its input or memory have been used successfully in deep learning models in recent years. Attention has improved image classification, image captioning, speech recognition, generative models, and learning algorithmic tasks, but it had probably the largest impact on neural machine translation. Recently, similar improvements have been obtained using alternative mechanisms that do not focus on a single part of a memory but operate on all of it in parallel, in a uniform way. Such mechanism, which we call active memory, improved over attention in algorithmic tasks, image processing, and in generative modelling. So far, however, active memory has not improved over attention for most natural language processing tasks, in particular for machine translation. We analyze this shortcoming in this paper and propose an extended model of active memory that matches existing attention models on neural machine translation and generalizes better to longer sentences. We investigate this model and explain why previous active memory models did not succeed. Finally, we discuss when active memory brings most benefits and where attention can be a better choice.

Citation Graph

Loading graph...

References [27]

Sort:

Filter:

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[3]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[4]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[5]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[6]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[7]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[8]Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

R. Williams - 1992

11 papers in library cite

It's alright for formalizing the concept, but it's a bit boring and doesn't add a lot from the middle on. Focuses too much in reviewing existing techniques and in stochastic units.

[9]Show, Attend and Tell: Neural Image Caption Generation With Visual Attention

K. Xu, Jimmy Lei Ba, R. Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, R. Zemel, Yoshua Bengio - 2015

12 papers in library cite

It's a nice paper. I liked the soft attention way more than the hard one, and I am a bit mad that it wasn't the best lol And also it's the first paper I read about multimodality, but it seems that this was bustling at the time. Also results are kinda bad.

[10]TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

M. Abadi, Akshat Agarwal, P. Barham, E. Brevdo, Ziru Chen, C. Citro, G. Corrado, A. Davis, Jeffrey Dean, M. Devin, Sanjay Ghemawat, I. Goodfellow, A. Harp, Geoffrey Irving, M. Isard, Y. Jia, R. Jozefowicz, Lukasz Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, Christopher Olah, M. Schuster, J. Shlens, B. Steiner, Ilya Sutskever, K. Talwar, P. Tucker, Vincent Vanhoucke, V. Vasudevan, F. Viegas, Oriol Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, Xiaoqiang Zheng - 2015

11 papers in library cite

This should be the golden standard to what framework papers should be. It's large, but it's not boring at all. Explains the core concepts while not going too deep as to describe unimportant things; explains design decisions and shortcomings... overall amazing

[11]On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Kyunghyun Cho, B. V. Merrienboer, D. Bahdanau, Yoshua Bengio - 2014

9 papers in library cite

This just builds upon other work, comparing and analyzing previous architectures. This faces a specific problem of the time: long sentences. Not too relevant today.

[12]Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

G. Dahl, D. Yu, L. Deng, Alex Acero - 2012

19 papers in library cite

Good paper, very well written and probably the best explanation of RBMs and DBNs I've seen. However, I don't see a lot of impact and seems very derivative from other works.

[13]Neural Turing Machines

Alex Graves, G. Wayne, Ivo Danihelka - 2014

18 papers in library cite

This paper is amazing. If someone told me that NNs could use and address memory by position I wouldn't believe it worked. Very nice, but it's a shame that it's just a toy example.

[14]DRAW: A Recurrent Neural Network for Image Generation

K. Gregor, Ivo Danihelka, Alex Graves, D. J. Rezende, Daan Wierstra - 2015

5 papers in library cite

This is SO cool! So interesting that it was biologically inspired. First actual image generation that seems to work, also a jump from the RBM/Autoencoder stuff, and the incremental drawing is amazing!

[15]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[16]Grammar as a Foreign Language

Geoffrey Hinton - 2015

9 papers in library cite

It's a nice paper showing that attention can be used for parsing. However, parsing is boring and is very derivative. Good paper nonetheless.

[17]Inferring Algorithmic Patterns With Stack-Augmented Recurrent Nets

Armand Joulin, Tomas Mikolov - 2015

9 papers in library cite

Very underwhelming TBH. I expected more after reading the Neural Turing Machine paper. This reads like "yeah, we lost the race, here's what we were doing before they did something better"

[18]Neural GPUs Learn Algorithms

Lukasz Kaiser, Ilya Sutskever - 2016

5 papers in library cite

Results and architecture are nice but TBH it is very poorly written... Doesn't seem like a lot of impact either.

[19]Fast Algorithms for Convolutional Neural Networks

A. Lavin - 2015

3 papers in library cite

Optimization trick

[20]Modeling Coverage for Neural Machine Translation

Zhuowen Tu, Z. L. Lu, Yibo Liu, Xiaodong Liu, H. Li - 2016

4 papers in library cite

Beam search coverage study

[21]Grid Long Short-Term Memory

N. Kalchbrenner, Ivo Danihelka, Alex Graves - 2016

3 papers in library cite

[22]One-Shot Generalization in Deep Generative Models

D. J. Rezende, S. Mohamed, Ivo Danihelka, K. Gregor, Daan Wierstra - 2016

2 papers in library cite

Generative, better than DRAW

[23]Towards Conceptual Compression

K. Gregor, F. Besse, D. J. Rezende, Ivo Danihelka, Daan Wierstra - 2016

1 paper in library cites

Apparently better than DRAW

[24]Encoding Source Language With Convolutional Neural Network for Machine Translation

Fanqing Meng, Z. L. Lu, Mingliang Wang, H. Li, W. Jiang, Qian Liu - 2015

3 papers in library cite

[25]Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

Q. Liao, T. Poggio - 2016

2 papers in library cite

[26]Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

X. Shi, Ziru Chen, Haiming Wang, D. Y. Yeung, W. K. Wong, W. C. Woo - 2015

2 papers in library cite

[27]Variable Rate Image Compression With Recurrent Neural Networks

G. Toderici, S. M. O'malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, R. Sukthankar - 2016

2 papers in library cite

Cited by

2

papers in your library

Cites

23

papers in your library

Read

on August 4, 2025

So nice to see an alternative that works as well as attention!

Tags

Paper Aliases

No aliases