Papperoni

2020

Learning to Summarize from Human Feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano

Open PDF Google Scholar

citations

Cite Score

AI summary

This paper introduces a method for training language models to optimize for human preferences, using a large dataset of human comparisons to train a reward model, which then fine-tunes a summarization policy with reinforcement learning, achieving state-of-the-art results on the Reddit TL;DR dataset and transferring to CNN/DM news articles.

Main Contributions

Demonstrated significant improvement in summary quality by training a model to optimize for human preferences using reinforcement learning.
Collected a large, high-quality dataset of human comparisons between summaries for training a reward model.
Showed that human feedback models significantly outperform larger supervised learning models and generalize well to new domains (CNN/DM news articles) without specific fine-tuning.
Conducted extensive empirical analyses to understand the reward model's generalization capabilities and its outperformance of ROUGE at predicting human preferences.
Publicly released the human feedback dataset for further research, containing 64,832 summary comparisons.

Abstract

As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about—summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts [63] and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles [22], producing summaries nearly as good as the human reference without any news-specific fine-tuning.2 We conduct extensive analyses to understand our human feedback dataset and fine-tuned models. We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.

Citation Graph

Loading graph...

References [73]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite