Papperoni

2018

Hierarchical Neural Story Generation

A. Fan, Martha Lewis, Yann Dauphin

Open PDF Google Scholar

citations

Cite Score

56

AI summary

The paper introduces a hierarchical story generation model using a large dataset of 300K human-written stories with writing prompts, model fusion, and gated multi-scale self-attention, achieving improvements over strong baselines in automated and human evaluations.

Main Contributions

Introduces a hierarchical story generation model.
Collects and releases a large dataset of 300K human-written stories paired with writing prompts.
Proposes a novel form of model fusion to improve the relevance of the story to the prompt.
Introduces a new gated multi-scale self-attention mechanism to model long-range context.
Achieves significant improvements over strong baselines on both automated and human evaluations.

Abstract

We explore story generation: creative systems that can build coherent and fluent passages of text about a topic. We collect a large dataset of 300K human-written stories paired with writing prompts from an online forum. Our dataset enables hierarchical story generation, where the model first generates a premise, and then transforms it into a passage of text. We gain further improvements with a novel form of model fusion that improves the relevance of the story to the prompt, and adding a new gated multi-scale self-attention mechanism to model long-range context. Experiments show large improvements over strong baselines on both automated and human evaluations. Human judges prefer stories generated by our approach to those from a strong non-hierarchical model by a factor of two to one.

Citation Graph

Loading graph...

References [30]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[3]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[4]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[5]Enriching Word Vectors With Subword Information

Tomas Mikolov - 2017

7 papers in library cite

It's just word2vec with ngrams instead of full words. Unsurprising, but important nonetheless

[6]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[7]On the Importance of Initialization and Momentum in Deep Learning

Ilya Sutskever, James Martens, G. Dahl, Geoffrey Hinton - 2013

13 papers in library cite

They give very good context and it's easy to understand that they are doing this as a counterpoint to HF. Surprising results as well. I just think it was made obsolete by relu

[8]Convolutional Sequence to Sequence Learning

J. Gehring, Michael Auli, D. Grangier, D. Yarats, Yann Dauphin - 2017

3 papers in library cite

It's a good paper. I liked that it is very similar to the transformer methodology (but they don't use convolutions there). I need to go back and read the transformers paper to make parallels.

[9]A Neural Attention Model for Abstractive Sentence Summarization

Alexander M. Rush, S. Chopra, Jason Weston - 2015

13 papers in library cite

TBH the paper is a bit boring and nothing new after reading a bunch of more modern techniques. I feel that they could have done a better job considering that seq2seq existed at the time. Either way, points for being the first to propose summarization with NNs.

[10]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[11]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[12]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[13]Generating Wikipedia by Summarizing Long Sequences

P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, Lukasz Kaiser, Noam Shazeer - 2018

7 papers in library cite

Very nice, the first ones that realized that they could use decoder only from Transformers. Also nice that they expanded the context window, and got nice results. Very inventive to use citations from wikipedia and the first paragraph as target.

[14]Unsupervised Pretraining for Sequence to Sequence Learning

P. Ramachandran, P. J. Liu, Quoc V. Le - 2017

9 papers in library cite

It's alright, but it's the same Seq2Seq thing with pretraining

[15]Billion-Scale Similarity Search With Gpus

J. Johnson, M. Douze, Hervé Jégou - 2017

4 papers in library cite

Efficient KNN with GPUs

[16]Language Modeling With Gated Convolutional Networks

Yann N. Dauphin, A. Fan, Michael Auli, D. Grangier - 2016

8 papers in library cite

[17]Challenges in Data-to-Document Generation

Sam Wiseman, S. Shieber, A. Rush - 2017

2 papers in library cite

[18]On Using monolingual corpora in Neural Machine Translation

C. G. Gulcehre, O. Firat, K. Xu, Kyunghyun Cho, L. Barrault, H. C. Lin, F. Bougares, Holger Schwenk, Yoshua Bengio - 2015

3 papers in library cite

Monolingual dataset for MT?

[19]Cold Fusion: Training Seq2seq Models Together With Language Models

A. Sriram, Heewoo Jun, S. Satheesh, A. Coates - 2017

1 paper in library cites

Joint learning of LMs and seq2seq

[20]A Diversity-Promoting Objective Function for Neural Conversation Models

Jeffrey Li, M. Galley, Chris Brockett, Jianfeng Gao, B. Dolan - 2016

4 papers in library cite

[21]A Hierarchical Neural Autoencoder for Paragraphs and Documents

Jeffrey Li, M. Luong, Dan Jurafsky - 2015

2 papers in library cite

[22]Chinese Poetry Generation With Recurrent Neural Networks

X. Zhang, Mirella Lapata - 2014

2 papers in library cite

[23]Diverse Beam Search: Decoding Diverse Solutions From Neural Sequence Models

A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. J. Crandall, D. Batra - 2016

2 papers in library cite

[24]Event Representations for Automated Story Generation With Deep Neural Nets

L. Martin, P. Ammanabrolu, W. Hancock, Shivalika Singh, B. Harrison, M. Riedl - 2017

1 paper in library cites

[25]Generating Long and Diverse Responses With Neural Conversation Models

L. Shao, S. Gouws, D. Britz, Anna Goldie, B. Strope, R. Kurzweil - 2017

1 paper in library cites

[26]Hierarchical Text Generation and Planning for Strategic Dialogue

D. Yarats, Martha Lewis - 2017

1 paper in library cites

[27]Story Generation From Sequence of Independent Short Descriptions

P. Jain, P. Agrawal, A. Mishra, M. Sukhwani, A. Laha, K. Sankararayanan - 2017

1 paper in library cites

[28]Toward Automated Story Generation With markov chain monte Carlo Methods and Deep Neural Networks

B. Harrison, C. Purdy, M. Riedl - 2017

1 paper in library cites

[29]Towards Better Decoding and Language Model Integration in Sequence to Sequence Models

J. Chorowski, Navdeep Jaitly - 2016

1 paper in library cites

[30]Writing Stories With Help From Recurrent Neural Networks

M. Roemmele - 2016

1 paper in library cites

Cited by

4

papers in your library

Cites

19

papers in your library

Read

on November 10, 2025

I had higher hopes for this paper. The methodology is nice but at this stage things start to get waaay to complicated

Tags

Paper Aliases

No aliases