Papperoni

2025

Chain of Thought Monitorability: A New and Fragile Opportunity for Al Safety

Ilya Sutskever

Open PDF Google Scholar

citations

AI summary

This paper introduces chain of thought (CoT) monitoring as a method for AI safety by monitoring AI systems' reasoning in natural language. It recommends further research and investment in CoT monitoring while also recognizing its limitations and potential fragility.

Main Contributions

Introduces chain of thought (CoT) monitoring as a new method for AI safety.
Recommends further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods.
Highlights the potential fragility of CoT monitorability and recommends that frontier model developers consider the impact of development decisions on it.
Discusses the limitations of CoT monitoring.
Encourages the research community and frontier AI developers to make the best use of CoT monitorability and study how it can be preserved.

Abstract

AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.

Citation Graph

Loading graph...

References [42]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]Deepseek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Deepseek Ai - 2025

2 papers in library cite

Finally the RLVR paper!

[3]Let's Verify Step by Step

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe - 2023

4 papers in library cite

Fun read but nothing really surprising in terms of methodology. Just proving PRMs are better than ORM (again, not surprising)

[4]Solving Math Word Problems With Process- And Outcome-Based Feedback

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins - 2022

4 papers in library cite

Very low 3. Almost 2. Boring, too big, and nothing surprising. Results were proved (partially) wrong later. I think they just formalized ORM vs. PRM

[5]Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei, Xinpeng Wang, Dale Schuurmans, Maarten Bosma, Fanyue Xia, E. Chi, Quoc V. Le, Denny Zhou - 2022

10 papers in library cite

CoT

[6]Large Language Models Are Zero-Shot Reasoners

T. Kojima, Shixiang Shane Gu, M. Reid, Y. Matsuo, Y. Iwasawa - 2022

6 papers in library cite

REASONING!

[7]Qwen3 Technical Report

A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, Bo Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, Haowei Lin, Jie Tang, Jihan Yang, J. Tu, J. Zhang, Jihan Yang, Jihan Yang, Jingren Zhou, Jingren Zhou, Junyang Lin, K. Dang, K. Bao, K. Yang, Longhui Yu, L. Deng, M. Li, M. Xue, M. Li, Peizhao Zhang, Peng Wang, Qihao Zhu, R. Men, R. Gao, Shuming Liu, S. Luo, Tao Li, T. Tang, W. Yin, Xiang Ren, Xinpeng Wang, X. Zhang, Xiang Ren, Yu Fan, Yu Su, Y. Z. Zhang, Y. Z. Zhang, Y. Wan, Yibo Liu, Zhengtao Wang, Z. Cui, Zhengyou Zhang, Zijian Zhou, Z. Qiu - 2025

5 papers in library cite

[8]Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

Laria Reynolds, Kyle Mcdonell - 2021

1 paper in library cites

CoT

[9]Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

M. Turpin, J. Michael, Ethan Perez, Samuel R. Bowman - 2023

2 papers in library cite

[10]Open Problems and Fundamental Limitations of Reinforcement Learning From Human Feedback

S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, Tomasz Korbak, D. Lindner, P. Freire, T. T. Wang, S. Marks, C. R. Segerie, M. Carroll, A. Peng, P. J. K. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Biyik, A. D. Dragan, David Krueger, D. Sadigh, D. H. Menell - 2023

1 paper in library cites

[11]OpenAI o1 System Card

Openai - 2024

1 paper in library cites

[12]Training Large Language Models to Reason in a Continuous Latent Space

S. Hao, S. Sukhbaatar, D. Su, Xiang Lisa Li, Z. Hu, Jason Weston, Yuandong Tian - 2024

2 papers in library cite

Reasoning in latent space

[13]Measuring Faithfulness in Chain-of-Thought Reasoning

Tamera Lanham, Anna Chen, A. Radhakrishnan, B. Steiner, C. Denison, Danny Hernandez, Dustin Li, Esin Durmus, E. Hubinger, Jackson Kernion, K. Lukosiute, K. Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shusheng Yang, Tom Henighan, Timothy Maxwell, Timothy Telleen Lawton, Tristan Hume, Zac Hatfield Dodds, Jared Kaplan, J. Brauner, Samuel R. Bowman, Ethan Perez - 2023

2 papers in library cite

[14]Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

Bowen Baker, J. Huizinga, Leo Gao, Z. Dou, M. Y. Guan, A. Madry, Wojciech Zaremba, J. Pachocki, D. Farhi - 2025

2 papers in library cite

[15]Reasoning Models Don't Always Say What They Think

Yanru Chen, J. Benton, A. Radhakrishnan, Jonathan Uesato, C. Denison, John Schulman, A. Somani, P. Hase, M. Wagner, F. Roger, V. Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, Ethan Perez - 2025

2 papers in library cite

[16]Chain-of-Thought Reasoning in the Wild Is Not Always Faithful

I. Arcuschin, J. Janiak, R. Krzyzanowski, S. Rajamanoharan, Neel Nanda, A. Conmy - 2025

2 papers in library cite

[17]When Chain of Thought Is Necessary, Language Models Struggle to Evade Monitors

S. Emmons, E. Jenner, D. K. Elson, Rif A. Saurous, S. Rajamanoharan, H. Chen, I. Shafkat, R. Shah - 2025

2 papers in library cite

[18]On the Biology of a Large Language Model

J. Lindsey, W. Gurnee, E. Ameisen, Berlin Chen, A. Pearce, N. L. Turner, C. Citro, D. Abrahams, S. Carter, B. Hosmer, J. Marcus, M. Sklar, A. Templeton, T. Bricken, C. Mcdougall, H. Cunningham, Tom Henighan, A. Jermyn, Andy Jones, A. Persic, Z. Qi, T. B. Thompson, S. Zimmerman, K. Rivoire, Tom Conerly, Christopher Olah, J. Batson - 2025

2 papers in library cite

Anthropic. Not a paper, but seems extremely nice

[19]Claude 3.7 Sonnet System Card

Anthropic - 2024

1 paper in library cites

[20]On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models With No Catastrophic Forgetting

Tomasz Korbak, H. Elsahar, German Kruszewski, M. Dymetmant - 2022

2 papers in library cite

[21]Scaling Up Test-Time Compute With Latent Reasoning: A Recurrent Depth Approach

J. Geiping, S. Mcleish, N. Jain, J. Kirchenbauer, Shivalika Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, T. Goldstein - 2025

2 papers in library cite

[22]A Framework for Evaluating Emerging Cyberattack Capabilities of AI

M. Rodriguez, R. A. Popa, F. Flynn, L. Liang, A. Dafoe, A. Wang - 2025

1 paper in library cites

[23]AI Control: Improving Safety Despite Intentional Subversion

R. Greenblatt, Buck Shlegeris, K. Sachan, F. Roger - 2024

1 paper in library cites

[24]Alignment Faking in Large Language Models

R. Greenblatt, C. Denison, B. Wright, F. Roger, M. Macdiarmid, S. Marks, J. Treutlein, T. Belonax, Jixuan Chen, David Duvenaud, A. Khan, J. Michael, S. Mindermann, Ethan Perez, L. Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, E. Hubinger - 2024

1 paper in library cites

[25]An Approach to Technical Agi Safety and Security

R. Shah, A. Irpan, A. M. Turner, A. Wang, A. Conmy, D. Lindner, J. B. Cohen, L. Ho, Neel Nanda, R. A. Popa, R. Jain, R. Greig, S. Albanie, S. Emmons, S. Farquhar, S. Krier, S. Rajamanoharan, S. Bridgers, T. Ijitoe, Tom Everitt, V. Krakovna, V. Varma, V. Mikulik, Z. Kenton, D. Orr, Shane Legg, N. Goodman, A. Dafoe, F. Flynn, A. D. Dragan - 2025

1 paper in library cites

[26]Auditing Language Models for Hidden Objectives

S. Marks, J. Treutlein, T. Bricken, J. Lindsey, J. Marcus, S. M. Sharma, D. Ziegler, E. Ameisen, J. Batson, T. Belonax, Samuel R. Bowman, S. Carter, Berlin Chen, H. Cunningham, C. Denison, F. Dietz, S. Golechha, A. Khan, J. Kirchner, Jan Leike, A. Meek, K. N. Gasparian, E. Ong, Christopher Olah, A. Pearce, F. Roger, J. Salle, A. Shih, Meg Tong, D. Thomas, K. Rivoire, A. Jermyn, M. Macdiarmid, Tom Henighan, E. Hubinger - 2025

1 paper in library cites

[27]Chain of Thought Empowers Transformers to Solve Inherently Serial Problems

Zhiyuan Li, Haozhe Liu, Denny Zhou, T. Ma - 2024

1 paper in library cites

[28]Claude Sonnet 3.7 (Often) Knows When It's in Alignment Evaluations

N. G. Dill, M. Balesni, J. Scheurer, M. Hobbhahn - 2025

1 paper in library cites

[29]Constitutional Classifiers: Defending Against Universal Jailbreaks Across Thousands of Hours of Red Teaming

Mrinank Sharma, Meg Tong, J. Mu, Jason Wei, J. Kruthoff, S. Goodfriend, E. Ong, A. Peng, R. Agarwal, C. Anil, Amanda Askell, N. Bailey, J. Benton, E. Bluemke, Samuel R. Bowman, E. Christiansen, H. Cunningham, A. Dau, A. Gopal, R. Gilson, L. Graham, L. Howard, N. Kalra, Teddy Lee, K. Lin, P. Lofgren, F. Mosconi, C. O'hara, Catherine Olsson, L. Petrini, S. Rajani, N. Saxena, A. Silverstein, T. Singh, T. Sumers, L. Tang, K. K. Troy, C. Weisser, R. Zhong, G. Zhou, Jan Leike, Jared Kaplan, Ethan Perez - 2025

1 paper in library cites

[30]Cot Red-Handed: Stress Testing Chain-of-Thought Monitoring

B. Arnav, P. B. Perez, N. H. Burger, T. Kostolansky, H. Whittingham, M. Phuong - 2025

1 paper in library cites

[31]CTRL-Z: Controlling AI Agents via Resampling

A. Bhatt, C. Rushing, A. Kaufman, T. Tracy, V. Georgiev, D. Matolcsi, A. Khan, Buck Shlegeris - 2025

1 paper in library cites

[32]Distill Paraphrases

F. Roger - 2025

1 paper in library cites

[33]Evaluating Frontier Models for Stealth and Situational Awareness

M. Phuong, R. S. Zimmermann, Zhengtao Wang, D. Lindner, V. Krakovna, S. Cogan, A. Dafoe, L. Ho, R. Shah - 2025

1 paper in library cites

[34]Frontier Models Are Capable of in-Context Scheming

A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, M. Hobbhahn - 2024

1 paper in library cites

[35]Guidelines for Capability Elicitation

Metr - 2024

1 paper in library cites

[36]Introducing Docent

K. Meng, V. Huang, Jacob Steinhardt, S. Schwettmann - 2025

1 paper in library cites

[37]Llamafirewall: An Open Source Guardrail System for Building Secure AI Agents

S. Chennabasappa, C. Nikolaidis, Dawn Song, D. Molnar, S. Ding, S. Wan, S. Whitman, L. Deason, N. Doucette, A. Montilla, A. Gampa, B. D. Paola, D. Gabi, J. Crnkovich, J. C. Testud, K. He, R. Chaturvedi, W. Zhou, J. Saxe - 2025

1 paper in library cites

[38]Multi-Agent Communication Meets Natural Language: Synergies Between Functional and Structural Language Learning

A. Lazaridou, A. Potapenko, O. Tieleman - 2020

1 paper in library cites

[39]Open Problems in Mechanistic Interpretability

L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, Jeffrey Wu, L. Bushnaq, N. G. Dill, S. Heimersheim, A. Ortega, J. Bloom, Stella Biderman, Adria Garriga Alonso, A. Conmy, Neel Nanda, J. Rumbelow, M. Wattenberg, N. Schoots, John Miller, E. J. Michaud, S. Casper, M. Tegmark, William Saunders, D. Bau, E. Todd, A. Geiger, Mor Geva, J. Hoogland, D. Murfet, T. Mcgrath - 2025

1 paper in library cites

[40]Replibench: Evaluating the Autonomous Replication Capabilities of Language Model Agents

S. Black, A. C. Stickland, J. Pencharz, O. Sourbut, M. Schmatz, J. Bailey, O. Matthews, B. Millwood, A. Remedios, A. Cooney - 2025

1 paper in library cites

[41]Sabotage Evaluations for Frontier Models

J. Benton, M. Wagner, E. Christiansen, C. Anil, Ethan Perez, J. Srivastav, Esin Durmus, Deep Ganguli, Shauna Kravec, Buck Shlegeris, Jared Kaplan, H. Karnofsky, E. Hubinger, R. Grosse, Samuel R. Bowman, David Duvenaud - 2024

1 paper in library cites

[42]Us aisi and uk Aisi Joint Pre-Deployment Test: Anthropic's Claude 3.5 Sonnet

U. A. S. Institute, U. A. S. Institute - 2024

1 paper in library cites

Cited by

0

papers in your library

Cites

19

papers in your library

Read

on July 21, 2025

Hmm... It's a good push but really nothing new in terms of ideas. It's good to push horizons and it's very needed.

Tags

Paper Aliases

No aliases