Papperoni

2016

Long Short-Term Memory-Networks for Machine Reading

Mirella Lapata

Open PDF Google Scholar

citations

Cite Score

51

AI summary

This paper introduces the Long Short-Term Memory-Network (LSTMN), a machine reading simulator that extends the LSTM architecture with a memory network and attention mechanism. It achieves comparable or better performance than state-of-the-art models on language modeling, sentiment analysis, and natural language inference tasks.

Main Contributions

Introduces the Long Short-Term Memory-Network (LSTMN) for machine reading.
Replaces the single memory cell of an LSTM with a memory network, enabling adaptive memory usage and induction of relations among tokens.
Demonstrates how to integrate the LSTMN with an encoder-decoder architecture.
Achieves performance comparable or better to state-of-the-art models on language modeling, sentiment analysis, and natural language inference.
Introduces an intra-attention mechanism that induces undirected relations among tokens and optimizes the entire network in downstream tasks.

Abstract

In this paper we address the question of how to render sequence-level networks better at handling structured input. We propose a machine reading simulator which processes text incrementally from left to right and performs shallow reasoning with memory and attention. The reader extends the Long Short-Term Memory architecture with a memory network in place of a single memory cell. This enables adaptive memory usage during recurrence with neural attention, offering a way to weakly induce relations among tokens. The system is initially designed to process a single sequence but we also demonstrate how to integrate it with an encoder-decoder architecture. Experiments on language modeling, sentiment analysis, and natural language inference show that our model matches or outperforms the state of the art.

Citation Graph

Loading graph...

References [50]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[4]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[5]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[6]Convolutional Neural Networks for Sentence Classification

Yoon Kim - 2014

8 papers in library cite

It's nice, goes straight to the point. I can see why it has tons of citations. However, I am not sure it was as impactful as 20k citations.

[7]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[8]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[9]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[10]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[11]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[12]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[13]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[14]A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner - 2014

7 papers in library cite

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

[15]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[16]A Neural Attention Model for Abstractive Sentence Summarization

Alexander M. Rush, S. Chopra, Jason Weston - 2015

13 papers in library cite

TBH the paper is a bit boring and nothing new after reading a bunch of more modern techniques. I feel that they could have done a better job considering that seq2seq existed at the time. Either way, points for being the first to propose summarization with NNs.

[17]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[18]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[19]Memory Networks

Jason Weston, S. Chopra, Antoine Bordes - 2015

18 papers in library cite

The first half of the paper (when they discuss the concept in a very abstract way) is amazing. However, the actual methodology was very convoluted - I did not like it. I thought that Neural Turing Machines were inspired in this, but actually they are contemporary... So anyway, the concept is nice, execution is not.

[20]Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Richard Socher, Eric H. Huang, J. Pennin, C. Manning, A. Ng - 2011

10 papers in library cite

Good paper overall, but seems very simple ideas and complex implementation. Overall not very impactful

[21]Reasoning About Entailment With Neural Attention

Tim Rocktaschel, Edward Grefenstette, K. Hermann, T. Kocisky, Phil Blunsom - 2016

5 papers in library cite

It's nice that they are SotA on top of SNLI, but they just apply existing methodologies.

[22]Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

K. S. Tai, Richard Socher, Christopher D. Manning - 2015

6 papers in library cite

Baseline for skip-thoughts paper

[23]Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, Victor Zhong, R. Paulus, Richard Socher - 2015

9 papers in library cite

[24]Gated Feedback Recurrent Neural Networks

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2015

3 papers in library cite

Youshua Bengio + Gated Networks

[25]A Clockwork RNN

J. Koutnik, K. Greff, Faustino Gomez, Jürgen Schmidhuber - 2014

4 papers in library cite

Interesting title, not well cited... I added because it's schmidhuber

[26]Learning to Execute

Wojciech Zaremba, Ilya Sutskever - 2014

8 papers in library cite

They try to execute python code

[27]Learning Natural Language Inference With LSTM

Shijie Wang, J. J. Jiang - 2016

3 papers in library cite

NLI with LSTM - used as a baseline in the decomposable attention paper

[28]Learning Longer Memory in Recurrent Neural Networks

Tomas Mikolov, Armand Joulin, S. Chopra, M. Mathieu, Marc'aurelio Ranzato - 2015

8 papers in library cite

RNNs + longer memory

[29]Untersuchungen zu dynamischen neuronalen netzen

Sepp Hochreiter - 1991

18 papers in library cite

[30]A Fast Unified Model for Parsing and Sentence Understanding

S. Bowman, J. Gauthier, Abhinav Rastogi, R. Gupta, C. Manning, Christopher Potts - 2016

5 papers in library cite

[31]Dynamic Memory Networks for Visual and Textual Question Answering

Caiming Xiong, S. Merity, Richard Socher - 2016

5 papers in library cite

[32]Learning Context-Free Grammars: Capabilities and Limitations of a Recurrent Neural Network With an External Stack Memory

S. Das, C. Giles, G. Sun - 1992

5 papers in library cite

[33]Learning to Transduce With Unbounded Memory

Edward Grefenstette, K. Hermann, M. Suleyman, Phil Blunsom - 2015

5 papers in library cite

[34]Transition-Based Dependency Parsing With Stack Long Short-Term Memory

C. Dyer, M. Ballesteros, W. Ling, A. Matthews, N. Smith - 2015

2 papers in library cite

[35]A Deep Memory-Based Architecture for Sequence-to-Sequence Learning

Fanqing Meng, Z. L. Lu, Zhuowen Tu, H. Li, Qian Liu - 2015

1 paper in library cites

[36]A Study of the Knowledge Base Requirements for Passing an Elementary Science Test

Peter Clark, P. Harrison, N. Balasubramanian - 2013

1 paper in library cites

[37]Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency

Dan Klein, C. Manning - 2004

1 paper in library cites

[38]Deep Recursive Neural Networks for Compositionality in Language

O. Irsoy, C. Cardie - 2014

1 paper in library cites

[39]Depth-Gated Recurrent Neural Networks

K. Yao, T. Cohn, K. Vylomova, K. Duh, C. Dyer - 2015

1 paper in library cites

[40]Eye Movements in Reading and Information Processing: 20 Years of Research

K. Rayner - 1998

1 paper in library cites

[41]Identifying Relations for Open Information Extraction

A. Fader, S. Soderland, Oren Etzioni - 2011

1 paper in library cites

[42]Insensitivity of the Human Sentence-Processing System to Hierarchical Structure

S. Frank, R. Bod - 2011

1 paper in library cites

[43]Integration of Visual and Linguistic Information in Spoken Language Comprehension

M. Tanenhaus, M. S. Knowlton, K. Eberhard, J. Sedivy - 1995

1 paper in library cites

[44]Learning to Compose Neural Networks for Question Answering

Jacob Andreas, M. Rohrbach, Trevor Darrell, Dan Klein - 2016

1 paper in library cites

[45]Locality and Parsing Complexity

L. Konieczny - 2000

1 paper in library cites

[46]Molding CNNS for Text: Non-Linear, Non-Consecutive Convolutions

T. Lei, R. Barzilay, T. Jaakkola - 2015

1 paper in library cites

[47]Open Information Extraction: The Second Generation

Oren Etzioni, A. Fader, J. Christensen, S. Soderland, Mausam - 2011

1 paper in library cites

[48]Recovery From Misanalyses of Garden-Path Sentences

F. Ferreira, J. Henderson - 1991

1 paper in library cites

[49]Recurrent Memory Network for Language Modeling

K. Tran, A. Bisazza, C. Monz - 2016

1 paper in library cites

[50]Unsupervised Ontology Induction From Text

H. Poon, P. Domingos - 2010

1 paper in library cites

Cited by

8

papers in your library

Cites

28

papers in your library

Read

on October 17, 2025

I read this more as an example of intra-attention, but this is not the main focus of the paper. I think visualization/explanation is a bit bad, and it doesn't seem too impactful. I kept thinking that this is starting to get too complicated, and indeed it was surpassed by transformers right after that.

Tags

Paper Aliases

No aliases