Papperoni

2020

Realtoxicityprompts: Evaluating Neural Toxic Degeneration in Language Models

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, Noah A. Smith

Open PDF Google Scholar

citations

Cite Score

55

AI summary

This paper introduces REALTOXICITYPROMPTS, a 100K-prompt dataset, to evaluate toxic degeneration in pretrained neural language models, revealing that LMs can generate toxic content even from non-toxic prompts, and that while some mitigation methods help, none are failsafe, highlighting issues with pretraining data.

Main Contributions

Introduction of REALTOXICITYPROMPTS, a dataset of 100K naturally occurring, sentence-level prompts with toxicity scores.
Empirical finding that pretrained LMs (GPT-1, GPT-2, GPT-3, CTRL) can degenerate into toxic text even from seemingly innocuous prompts.
Evaluation of controllable generation methods, showing that data- or compute-intensive methods are more effective but no current method is failsafe.
Analysis of pretraining corpora (OpenAI WebText and OpenWebText Corpus), identifying significant amounts of offensive and toxic content.
Recommendations for better data selection processes and public release of relevant information during data collection for pretraining corpora.

Abstract

Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration. We create and release REALTOXICITYPROMPTS, a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely-used toxicity classifier. Using REALTOXICITYPROMPTS, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts. We empirically assess several controllable generation methods, and find that while data- or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxicity than simpler solutions (e.g., banning "bad" words), no current method is failsafe against neural toxic degeneration. To pinpoint the potential cause of such persistent toxic degeneration, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Radford et al., 2019), and find a significant amount of offensive, factually unreliable, and otherwise toxic content. Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining.

Citation Graph

Loading graph...

References [83]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[4]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[5]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[6]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[7]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[8]Enriching Word Vectors With Subword Information

Tomas Mikolov - 2017

7 papers in library cite

It's just word2vec with ngrams instead of full words. Unsurprising, but important nonetheless

[9]Overcoming Catastrophic Forgetting in Neural Networks

J. Kirkpatrick, Razvan Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. G. Barwinska, Demis Hassabis, C. Clopath, D. Kumaran, Raia Hadsell - 2017

5 papers in library cite

Very nice, intuitive, and impressive results.

[10]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[11]Huggingface's Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, Sam Shleifer, P. V. Platen, C. Ma, Yacine Jernite, J. Plu, Chenfeng Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, Alexander M. Rush - 2019

7 papers in library cite

It's a solid paper, but these framework papers are usually boring... Yet, it's nice to see that they started with reduced scope (and how large it got now).

[12]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[13]Fine-Tuning Language Models From Human Preferences

Geoffrey Irving - 2020

7 papers in library cite

It's so simple how they do it, plus I absolutely LOVED the "challenges" section and how honest they were about it. This is true research!

[14]Pytorch: An Imperative Style, High-Performance Deep Learning Library

A. Paszke, S. Gross, Francisco Massa, Adam Lerer, J. Bradbury, G. Chanan, T. Killeen, Zongyu Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Devito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, Jinze Bai, S. Chintala - 2019

2 papers in library cite

[15]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

17 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

[16]The Curious Case of Neural Text Degeneration

Ari Holtzman, J. Buys, L. Du, M. Forbes, Yejin Choi - 2019

5 papers in library cite

[17]Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

Suchin Gururangan, A. Marasovic, Swabha Swayamdipta, K. Lo, I. Beltagy, D. Downey, Noah A. Smith - 2020

2 papers in library cite

[18]Model Cards for Model Reporting

M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, T. Gebru - 2018

5 papers in library cite

Seems like something foundational - we use at nubank!

[19]Ctrl: A Conditional Transformer Language Model for Controllable Generation

Nitish Shirish Keskar, B. Mccann, L. R. Varshney, Caiming Xiong, Richard Socher - 2019

4 papers in library cite

[20]Language (Technology) Is Power: A Critical Survey of "Bias" in NLP

S. L. Blodgett, S. Barocas, H. D. Iii, H. Wallach - 2020

7 papers in library cite

[21]Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science

E. M. Bender, B. Friedman - 2018

4 papers in library cite

[22]The Woman Worked as a Babysitter: On Biases in Language Generation

E. Sheng, K. W. Chang, P. Natarajan, Nanyun Peng - 2019

4 papers in library cite

[23]Neural Text Generation With Unlikelihood Training

S. Welleck, I. Kulikov, S. Roller, E. Dinan, Kyunghyun Cho, Jason Weston - 2019

3 papers in library cite

[24]Show Your Work: Improved Reporting of Experimental Results

J. Dodge, Suchin Gururangan, D. Card, Richard Schwartz, Noah A. Smith - 2019

3 papers in library cite

[25]spaCy 2: Natural Language Understanding With Bloom Embeddings, Convolutional Neural Networks and Incremental Parsing

M. Honnibal, I. Montani - 2017

3 papers in library cite

[26]Bringing the People Back In: Contesting Benchmark Machine Learning Datasets

E. Denton, A. Hanna, R. Amironesei, A. Smart, H. Nicole, M. K. Scheuerman - 2020

2 papers in library cite

[27]Build It Break It Fix It for Dialogue Safety: Robustness From Adversarial Human Attack

E. Dinan, S. Humeau, B. Chintagunta, Jason Weston - 2019

2 papers in library cite

[28]Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts

L. Breitfeller, E. Ahn, David Jurgens, Y. Tsvetkov - 2019

2 papers in library cite

[29]On Measuring Social Biases in Sentence Encoders

C. May, A. Wang, S. Bordia, Samuel R. Bowman, R. Rudinger - 2019

2 papers in library cite

[30]Openwebtext corpus

A. Gokaslan, V. Cohen - 2019

2 papers in library cite

[31]Plug and Play Language Models: A Simple Approach to Controlled Text Generation

S. Dathathri, Andrea Madotto, J. Lan, J. Hung, E. Frank, P. Molino, Jason Yosinski, Rosanne Liu - 2019

2 papers in library cite

[32]Social Bias Frames: Reasoning About Social and Power Implications of Language

Maarten Sap, S. Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, Yejin Choi - 2020

2 papers in library cite

[33]“Transforming" Delete, Retrieve, Generate Approach for Controlled Text Style Transfer

A. Sudhakar, B. Upadhyay, A. Maheswaran - 2019

1 paper in library cites

[34]#Gamergate and the Fappening: How Reddit's Algorithm, Governance, and Culture Support Toxic Technocultures

A. Massanari - 2017

1 paper in library cites

[35]A Computational Approach to Politeness With Application to Social Factors

C. D. N. Mizil, M. Sudhof, Dan Jurafsky, J. Leskovec, Christopher Potts - 2013

1 paper in library cites

[36]A Large Labeled Corpus for Online Harassment Research

J. Golbeck, Z. Ashktorab, R. O. Banjo, A. Berlinger, S. Bhagwan, C. Buntain, P. Cheakalos, A. A. Geller, Q. Gergory, R. K. Gnanasekaran, R. R. Gunasekaran, K. M. Hoffman, J. Hottle, V. Jienjitlert, S. Khare, R. Lau, M. J. Martindale, S. Naik, H. L. Nixon, P. Ramachandran, K. M. Rogers, L. Rogers, M. S. Sarin, G. Shahane, J. Thanki, P. Vengataraman, Z. Wan, D. M. Wu - 2017

1 paper in library cites

[37]Affect-Lm: A Neural Language Model for Customizable Affective Text Generation

Sayan Ghosh, M. Chollet, E. Laksana, Louis Philippe Morency, S. Scherer - 2017

1 paper in library cites

[38]African American English: A Linguistic Introduction, 8.3.2002 Edition Edition

L. Green - 2002

1 paper in library cites

[39]Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter

Z. Waseem - 2016

1 paper in library cites

[40]BERT Has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

A. Wang, Kyunghyun Cho - 2019

1 paper in library cites

[41]Communities: Participatory Design For, With and by Communities

C. Disalvo, A. Clement, V. Pipek - 2012

1 paper in library cites

[42]Controlling Linguistic Style Aspects in Neural Language Generation

J. Ficler, Y. Goldberg - 2017

1 paper in library cites

[43]Conversations Gone Awry: Detecting Early Signs of Conversational Failure

J. Zhang, J. Chang, C. D. N. Mizil, L. Dixon, Y. Hua, D. Taraborelli, N. Thain - 2018

1 paper in library cites

[44]Demographic Dialectal Variation in Social Media: A Case Study of African-American English

S. L. Blodgett, L. Green, B. O'connor - 2016

1 paper in library cites

[45]Detecting Harassment in Real-Time as Conversations Develop

W. Stoop, F. Kunneman, A. V. D. Bosch, B. Miller - 2019

1 paper in library cites

[46]Discovering and Categorising Language Biases in Reddit

X. F. Aran, T. V. Nuenen, J. M. Such, N. Criado - 2020

1 paper in library cites

[47]Empathy, Ways of Knowing, and Interdependence as Mediators of Gender Differences in Attitudes Toward Hate Speech and Freedom of Speech

G. Cowan, D. Khatchadourian - 2003

1 paper in library cites

[48]Evaluating the Underlying Gender Bias in Contextualized Word Embeddings

C. Basta, M. R. C. Jussa, N. Casas - 2019

1 paper in library cites

[49]From User-Centered to Participatory Design Approaches

E. Sanders - 2002

1 paper in library cites

[50]Gender Bias in Contextualized Word Embeddings

J. Zhao, Tianle Wang, M. Yatskar, R. Cotterell, V. Ordonez, K. W. Chang - 2019

1 paper in library cites

[51]Gmail Smart Compose: Real-Time Assisted Writing

M. X. Chen, B. N. Lee, G. Bansal, Yue Cao, S. Zhang, J. Y. Lu, J. Tsay, Yuzhi Wang, A. M. Dai, Ziru Chen, T. Sohn, Yonghui Wu - 2019

1 paper in library cites

[52]How Automated Tools Discriminate Against Black Language

A. Chung - 2019

1 paper in library cites

[53]Know thy corpus! Robust Methods for Digital Curation of Web corpora

S. Sharoff - 2020

1 paper in library cites

[54]Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior

A. M. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, N. Kourtellis - 2018

1 paper in library cites

[55]Learning to Write With Cooperative Discriminators

Ari Holtzman, J. Buys, M. Forbes, Antoine Bosselut, D. Golub, Yejin Choi - 2018

1 paper in library cites

[56]Lessons From Archives: Strategies for Collecting Sociocultural Data in Machine Learning

E. S. Jo, T. Gebru - 2020

1 paper in library cites

[57]Measurement and Fairness

A. Z. Jacobs, H. M. Wallach - 2019

1 paper in library cites

[58]Measuring and Mitigating Unintended Bias in Text Classification

L. Dixon, Jeffrey Li, J. S. Sorensen, N. Thain, L. Vasserman - 2018

1 paper in library cites

[59]Measuring Bias in Contextualized Word Representations

K. Kurita, N. Vyas, A. Pareek, A. W. Black, Y. Tsvetkov - 2019

1 paper in library cites

[60]Measuring the Reliability of Hate Speech Annotations: The Case of the european refugee Crisis

B. Ross, M. Rist, G. Carbonell, B. Cabrera, N. Kurowsky, M. Wojatzki - 2017

1 paper in library cites

[61]Mining of Massive Datasets

A. Rajaraman, J. D. Ullman - 2011

1 paper in library cites

[62]Multi-Dimensional Gender Bias Classification

E. Dinan, A. Fan, L. Y. Wu, Jason Weston, Douwe Kiela, A. Williams - 2020

1 paper in library cites

[63]One-Step and Two-Step Classification for Abusive Language Detection on Twitter

J. H. Park, Pascale Fung - 2017

1 paper in library cites

[64]PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction

X. Ma, Maarten Sap, Hannah Rashkin, Yejin Choi - 2020

1 paper in library cites

[65]Predicting Factuality of Reporting and Bias of News Media Sources

R. Baly, G. Karadzhov, D. Alexandrov, J. Glass, P. Nakov - 2018

1 paper in library cites

[66]Preemptive Toxic Language Detection in Wikipedia Comments Using Thread-Level Context

M. Karan, J. Snajder - 2019

1 paper in library cites

[67]Quick, Community-Specific Learning: How Distinctive Toxicity Norms Are Maintained in Political Subreddits

A. Rajadesingan, P. Resnick, C. Budak - 2020

1 paper in library cites

[68]Racial Bias in Hate Speech and Abusive Language Detection Datasets

T. Davidson, D. Bhattacharya, I. Weber - 2019

1 paper in library cites

[69]Reddit Just Banned One of Its Most Toxic Forums. But It Won't Touch The_Donald

A. Romano - 2017

1 paper in library cites

[70]Seven-in-Ten Reddit Users Get News on the Site

M. Barthel, G. Stocking, J. Holcomb, A. Mitchell - 2016

1 paper in library cites

[71]Shirtless and Dangerous: Quantifying Linguistic Signals of Gender Bias in an Online Fiction Writing Community

E. Fast, T. Vachovsky, M. S. Bernstein - 2016

1 paper in library cites

[72]Social Biases in NLP Models as Barriers for Persons With Disabilities

B. Hutchinson, V. Prabhakaran, E. Denton, K. Webster, Y. Zhong, S. Denuy - 2020

1 paper in library cites

[73]Sparse Additive Generative Models of Text

J. Eisenstein, A. Ahmed, E. P. Xing - 2011

1 paper in library cites

[74]Talk to Transformer

A. King - 2019

1 paper in library cites

[75]The Impact of Toxic Language on the Health of Reddit Communities

S. Mohan, A. Guha, M. Harris, F. Popowich, A. Schuster, C. Priebe - 2017

1 paper in library cites

[76]The Problem With Bias: Allocative Versus Representational Harms in Machine Learning

S. Barocas, K. Crawford, A. Shapiro, H. Wallach - 2017

1 paper in library cites

[77]The Radicalization Risks of GPT-3 and Advanced Neural Language Models

K. Mcguffie, A. Newhouse - 2020

1 paper in library cites

[78]The Risk of Racial Bias in Hate Speech Detection

Maarten Sap, D. Card, S. Gabriel, Yejin Choi, Noah A. Smith - 2019

1 paper in library cites

[79]The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Nicholas Carlini, C. L. Liu, U. Erlingsson, J. Kos, D. X. Song - 2019

1 paper in library cites

[80]Transforming Classifier Scores Into Accurate Multiclass Probability Estimates

B. Zadrozny, C. Elkan - 2002

1 paper in library cites

[81]Understanding Black-Box Predictions via Influence Functions

P. W. Koh, Percy Liang - 2017

1 paper in library cites

[82]Universal Adversarial Triggers for Attacking and Analyzing NLP

E. Wallace, S. Feng, N. Kandpal, Matt Gardner, Shivalika Singh - 2019

1 paper in library cites

[83]Value Sensitive Design and Information Systems

B. Friedman, P. H. Kahn, A. Borning - 2008

1 paper in library cites

Cited by

1

papers in your library

Cites

19

papers in your library

Read

on May 30, 2026

Meh. Good push and good analysis but nothing exciting

Tags

Paper Aliases

No aliases