Papperoni

2022

Scaling Laws for Reward Model Overoptimization

Leo Gao, John Schulman, Jacob Hilton

citations

Cite Score

AI summary

This paper investigates overoptimization in reward models using a synthetic setup with a "gold-standard" reward model to study scaling laws for gold reward score changes, finding distinct functional forms for reinforcement learning and best-of-n sampling that scale smoothly with reward model parameters.

Main Contributions

Introduces a synthetic setup using a "gold-standard" reward model to study overoptimization, bypassing the high cost of human preference data.
Derives empirically validated functional forms (scaling laws) for how the gold reward model score changes when optimizing against a proxy reward model using either best-of-n sampling or reinforcement learning.
Observes that the coefficients in these scaling laws vary smoothly with the number of proxy reward model parameters, allowing for prediction of attained gold RM score.
Finds that larger policies benefit less from optimization but do not overoptimize more, and the gold scores peak at similar KL distances across policy sizes.
Demonstrates that a KL penalty in reinforcement learning acts similarly to early stopping, converging gold RM scores earlier but not improving the KLRL-gold reward frontier.

Abstract

In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed “gold-standard” reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-n sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.

Citation Graph

Loading graph...

References [55]

Sort:

Filter:

[1]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite