Papperoni

2016

Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond

R. Nallapati, B. Zhou, C. N. D. Santos, C. G. Gulcehre, Bing Xiang

Open PDF Google Scholar

citations

Cite Score

65

AI summary

This paper introduces a new abstractive text summarization model using Attentional Encoder-Decoder Recurrent Neural Networks, achieving state-of-the-art performance on two corpora. It proposes novel models addressing critical summarization problems and introduces a new dataset of multi-sentence summaries, establishing performance benchmarks.

Main Contributions

Applied an off-the-shelf attentional encoder-decoder RNN to summarization, achieving state-of-the-art results on two English corpora.
Proposed novel models that address concrete problems in summarization, providing additional performance improvement.
Introduced a new dataset for abstractive summarization of documents into multiple sentences and established benchmarks.
Introduced a feature-rich encoder to capture keywords using POS tags, named entities, and TF and IDF statistics.
Developed a switching generator-pointer model to handle rare/unseen words by pointing to their location in the source document.

Abstract

In this work, we model abstractive text summarization using Attentional Encoder Decoder Recurrent Neural Networks, and show that they achieve state-of-the-art performance on two different corpora. We propose several novel models that address critical problems in summarization that are not adequately modeled by the basic architecture, such as modeling key-words, capturing the hierarchy of sentence-to word structure, and emitting words that are rare or unseen at training time. Our work shows that many of our proposed models contribute to further improvement in performance. We also propose a new dataset consisting of multi-sentence summaries, and establish performance benchmarks for further research.

Citation Graph

Loading graph...

References [32]

Sort:

Filter:

[1]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[2]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[3]Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2014

11 papers in library cite

It's a good paper but results are veeeery underwhelming.

[4]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[5]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[6]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[7]Pointer Networks

Oriol Vinyals, M. Fortunato, Navdeep Jaitly - 2015

10 papers in library cite

Cool concept. Nice that it works and can find good solutions for TSP.

[8]A Neural Attention Model for Abstractive Sentence Summarization

Alexander M. Rush, S. Chopra, Jason Weston - 2015

13 papers in library cite

TBH the paper is a bit boring and nothing new after reading a bunch of more modern techniques. I feel that they could have done a better job considering that seq2seq existed at the time. Either way, points for being the first to propose summarization with NNs.

[9]Long Short-Term Memory-Networks for Machine Reading

Mirella Lapata - 2016

8 papers in library cite

I read this more as an example of intra-attention, but this is not the main focus of the paper. I think visualization/explanation is a bit bad, and it doesn't seem too impactful. I kept thinking that this is starting to get too complicated, and indeed it was surpassed by transformers right after that.

[10]On Using Very Large Target Vocabulary for Neural Machine Translation

Yoshua Bengio - 2014

12 papers in library cite

It's nice, but it starts getting a bit into the realm of "yeah, that seems like a minor improvement". It's nice that they use the importance sampling stuff from the previous paper though - I thought it had completely vanished :)

[11]Addressing the Rare Word Problem in Neural Machine Translation

T. Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba - 2014

14 papers in library cite

The method was very poorly explained. It was also worse than a paper released sooner, and more complicated. Overall not that good.

[12]Sequence-to-Sequence RNNS for Text Summarization

R. Nallapati, Bing Xiang, B. Zhou - 2016

2 papers in library cite

Very well cited, but maybe the same as the other ones

[13]Sequence to Sequence Video to Text

S. Venugopalan, M. Rohrbach, J. Donahue, R. J. Mooney, Trevor Darrell, K. Saenko - 2015

1 paper in library cites

[14]End-to-End Attention-Based Large Vocabulary Speech Recognition

D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, Yoshua Bengio - 2015

2 papers in library cite

Large Vocab speeck recog + Bengio

[15]Abstractive Sentence Summarization With Attentive Recurrent Neural Networks

S. Chopra, Michael Auli, A. Rush, S. Harvard - 2016

5 papers in library cite

Summarization with attention

[16]Pointing the Unknown Words

C. G. Gulcehre, S. Ahn, R. Nallapati, B. Zhou, Yoshua Bengio - 2016

7 papers in library cite

Bengio, pointer networks

[17]Temporal Attention Model for Neural Machine Translation

B. Sankaran, H. Mi, Y. A. Onaizan, A. Ittycheriah - 2016

3 papers in library cite

Intra-attention

[18]A Hierarchical Neural Autoencoder for Paragraphs and Documents

Jeffrey Li, M. Luong, Dan Jurafsky - 2015

2 papers in library cite

[19]BBN/Umd at Duc-2004: Topiary

D. Zajic, B. J. Dorr, Richard Schwartz - 2004

2 papers in library cite

[20]Headline Generation Based on Statistical Translation

M. Banko, V. O. Mittal, M. J. Witbrock - 2000

2 papers in library cite

[21]Lcsts: A Large Scale Chinese Short Text Summarization Dataset

B. Hu, Qinlang Chen, F. Zhu - 2015

2 papers in library cite

[22]Overcoming the Lack of Parallel Data in Sentence Compression

K. Filippova, Y. Altun - 2013

2 papers in library cite

[23]Sentence Compression Beyond Word Deletion

T. Cohn, Mirella Lapata - 2008

2 papers in library cite

[24]Title Generation With Quasi-Synchronous Grammar

K. Woodsend, Y. Feng, Mirella Lapata - 2010

2 papers in library cite

[25]Automatic Text Summarization Using a Machine Learning Approach

J. L. Neto, A. A. Freitas, C. A. A. Kaestner - 2002

1 paper in library cites

[26]Extractive Summarization Using Supervised and Semi-Supervised Learning

K. Wong, M. Wu, Wentao Li - 2008

1 paper in library cites

[27]Graph-Based Keyword Extraction for Single-Document Summarization

M. Litvak, M. Last - 2008

1 paper in library cites

[28]Heads: Headline Generation as Sequence Prediction Using an Abstract Feature-Rich Space

C. A. Colmenares, M. Litvak, A. Mantrach, F. Silvestri - 2015

1 paper in library cites

[29]Lexrank: Graph-Based Lexical Centrality as Salience in Text Summarization

G. Erkan, D. R. Radev - 2004

1 paper in library cites

[30]Long Story Short ÂĂŞ Global Unsupervised Models for Keyphrase Based Meeting Summarization

B. Favre, K. Riedhammer, D. H. Tur - 2010

1 paper in library cites

[31]Neural Summarization by Extracting Sentences and Words

J. Cheng, Mirella Lapata - 2016

1 paper in library cites

[32]Self Reinforcement for Important Passage Retrieval

D. M. D. M. Joaco, P. Neto, A. Gershman, J. Carbonell, R. Ribeiro, L. Marujo - 2013

1 paper in library cites

Cited by

10

papers in your library

Cites

17

papers in your library

Read

on October 31, 2025

Cool that they use pointer switching, and they introduce the CNN/daily mail dataset for summarization, which is a nice insight.

Tags

Paper Aliases

No aliases