Papperoni

2018

Generating Wikipedia by Summarizing Long Sequences

P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, Lukasz Kaiser, Noam Shazeer

Open PDF Google Scholar

citations

Cite Score

39

AI summary

This paper introduces a new method for generating Wikipedia articles by using a two-stage extractive-abstractive summarization framework on a large, parallel dataset. It uses a decoder-only Transformer architecture to attend to long sequences, achieving strong results in ROUGE scores and human evaluations.

Main Contributions

Introduces a two-stage extractive-abstractive framework for generating Wikipedia articles.
Proposes a decoder-only Transformer architecture for abstractive summarization that can handle very long sequences.
Demonstrates the effectiveness of the proposed model in generating fluent and coherent multi-sentence paragraphs and even whole Wikipedia articles.
Achieves strong results in ROUGE scores and human evaluations compared to traditional encoder-decoder architectures.
Releases the URLs used in the experiments to encourage further research on large-scale summarization.

Abstract

We show that generating English Wikipedia articles can be approached as a multi-document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder-decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations.

Citation Graph

Loading graph...

References [20]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[3]ROUGE: A Package for Automatic Evaluation of Summaries

Chin Yew Lin - 2004

9 papers in library cite

Wow, very poorly written and badly explained. This is the exact opposite of BLEU

[4]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[5]Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation

Yonghui Wu, M. Schuster, Ziru Chen, Quoc V. Le, M. Norouzi, W. Macherey, M. Krikun, Yue Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. J. Johnson, Xiaodong Liu, Lukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, Wenyi Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, Oriol Vinyals, G. S. Corrado, M. Hughes, Jeffrey Dean - 2016

15 papers in library cite

It's a very good paper but TBH doesn't bring anything new other than joining a bunch of existing stuff. I think it ended up being foundational because it's Google and several people used it as a base for future research. Good contribution then :)

[6]A Neural Attention Model for Abstractive Sentence Summarization

Alexander M. Rush, S. Chopra, Jason Weston - 2015

13 papers in library cite

TBH the paper is a bit boring and nothing new after reading a bunch of more modern techniques. I feel that they could have done a better job considering that seq2seq existed at the time. Either way, points for being the first to propose summarization with NNs.

[7]Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Noam Shazeer, Azalia Mirhoseini, K. Maziarz, A. Davis, Quoc Le, Geoffrey Hinton, Jeffrey Dean - 2017

9 papers in library cite

It's nice, but there's an important section in the middle about batch sizes that I don't quite understand. Not sure if I am missing some background knowledge or if they explain it poorly, and seems foundational to their main method... Either way, I did understand the methodology of the paper, and they have nice results :)

[8]Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond

R. Nallapati, B. Zhou, C. N. D. Santos, C. G. Gulcehre, Bing Xiang - 2016

10 papers in library cite

Cool that they use pointer switching, and they introduce the CNN/daily mail dataset for summarization, which is a nice insight.

[9]A Deep Reinforced Model for Abstractive Summarization

R. Paulus, Caiming Xiong, Richard Socher - 2017

7 papers in library cite

It's nice that they introduce intra-attention and RL, but at this point I think a lot of the work in attention is very derivative.

[10]Abstractive Sentence Summarization With Attentive Recurrent Neural Networks

S. Chopra, Michael Auli, A. Rush, S. Harvard - 2016

5 papers in library cite

Summarization with attention

[11]English Gigaword

R. Parker, D. Graff, J. Kong, K. Chen, K. Maeda - 2011

5 papers in library cite

[12]Wikireading: A Novel Large-Scale Language Understanding Task Over Wikipedia

D. Hewlett, A. Lacoste, Llion Jones, Illia Polosukhin, A. Fandrianto, J. Han, M. Kelcey, D. Berthelot - 2016

4 papers in library cite

[13]DBpedia – A Large-Scale, Multilingual Knowledge Base Extracted From Wikipedia

J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P. N. Mendes, S. Hellmann, M. Morsey, P. V. Kleef, S. Auer - 2014

2 papers in library cite

[14]Neural Text Generation From Structured Data With Application to the Biography Domain

R. Lebret, D. Grangier, Michael Auli - 2016

2 papers in library cite

[15]Automatically Generating Wikipedia Articles: A Structure-Aware Approach

C. Sauper, R. Barzilay - 2009

1 paper in library cites

[16]Overview of duc 2005

H. T. Dang - 2005

1 paper in library cites

[17]Textrank: Bringing Order Into Text

R. Mihalcea, P. Tarau - 2004

1 paper in library cites

[18]The Impact of Frequency on Summarization

A. Nenkova, L. Vanderwende - 2005

1 paper in library cites

[19]The Pagerank Citation Ranking: Bringing Order to the Web

L. Page, S. Brin, R. Motwani, T. Winograd - 1999

1 paper in library cites

[20]Using Tf-Idf to Determine Word Relevance in Document Queries

J. Ramos - 2003

1 paper in library cites

Cited by

7

papers in your library

Cites

10

papers in your library

Read

on August 6, 2025

Very nice, the first ones that realized that they could use decoder only from Transformers. Also nice that they expanded the context window, and got nice results. Very inventive to use citations from wikipedia and the first paragraph as target.

Tags

Paper Aliases

No aliases