Papperoni

2023

Instruction-Following Evaluation for Large Language Models

Jingren Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, Le Hou

Open PDF Google Scholar

citations

Cite Score

41

AI summary

This paper introduces **Instruction-Following Eval (IFEval)**, a benchmark for evaluating **Large Language Models (LLMs)**' ability to follow **verifiable natural language instructions**, comprising 25 instruction types and 500 prompts, and demonstrates its utility by evaluating two widely available **LLMs**.

Main Contributions

Introduces IFEval, a new approach for evaluating LLMs' instruction-following proficiency.
Defines and focuses on “verifiable instructions” amenable to objective verification.
Identifies 25 types of verifiable instructions and constructs approximately 500 prompts, each containing one or more instructions.
Proposes a robust verification approach that considers commonly seen variations to alleviate false negatives.
Provides evaluation results for GPT-4 and PaLM 2 S as baselines, showing their performance across various instruction types.

Abstract

One core capability of Large Language Models (LLMs) is to follow natural language instructions. However, the evaluation of such abilities is not standardized: Human evaluations are expensive, slow, and not objectively reproducible, while LLM-based auto-evaluation is potentially biased or limited by the ability of the evaluator LLM. To overcome these issues, we introduce Instruction-Following Eval (IFEval) for large language models. IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of “verifiable instructions” such as “write in more than 400 words” and “mention the keyword of AI at least 3 times”. We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We show evaluation results of two widely available LLMs on the market. Our code and data can be found at https://github.com/google-research/google-research/tree/master/instruction_following_eval

Citation Graph

Loading graph...

References [27]

Sort:

Filter:

[1]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

25 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[2]Training Language Models to Follow Instructions With Human Feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, C. Wainwright, Pamela Mishkin, Chiyuan Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, Ryan Lowe - 2022

15 papers in library cite

No new research here. Only true contribution is scaling RLHF to GPT 3.

[3]Judging LLM-as-a-Judge With MT-Bench and Chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Ziyi Wu, Yonghao Zhuang, Zongyu Lin, Zhiyuan Li, Dustin Li, E. Xing - 2023

4 papers in library cite

Could be better if the data was described better. I feel the number of citations is just because of the "llm-as-a-judge" headline

[4]Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba - 2021

16 papers in library cite

Very nice read! Nothin new in the methodology but very thoughtful and thorough analysis.

[5]GPT-4 Technical Report

Openai - 2023

14 papers in library cite

GPT 4

[6]Palm: Scaling Language Modeling With Pathways

Aakanksha Chowdhery, S. Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, A. Roberts, P. Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann - 2023

7 papers in library cite

[7]Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Yiwei Li, Xinpeng Wang, Mostafa Dehghani, Siddhartha Brahma, A. Webson, Shixiang Shane Gu, Z. Dai, Mirac Suzgun, X. Chen, Aakanksha Chowdhery, A. C. Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, Gaurav Mishra, A. Yu, V. Zhao, Y. Huang, Andrew Dai, H. Yu, Slav Petrov, Ed H. Chi, Jeffrey Dean, Jacob Devlin, A. Roberts, Denny Zhou, Quoc V. Le, Jason Wei - 2022

3 papers in library cite

[8]Finetuned Language Models Are Zero-Shot Learners

Jason Wei, Maarten Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, Quoc V. Le - 2021

5 papers in library cite

Missing author list

[9]A Survey on Evaluation of Large Language Models

2023

3 papers in library cite

[10]Multitask Prompted Training Enables Zero-Shot Task Generalization

V. Sanh, A. Webson, Colin Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja - 2021

6 papers in library cite

[11]Can Large Language Models Be an Alternative to Human Evaluations?

C. H. Chiang, H. Y. Lee - 2023

3 papers in library cite

[12]Large Language Models Are Not Fair Evaluators

Peng Wang, Lei Li, L. C. Chen, D. Zhu, B. Lin, Yue Cao, Qian Liu, T. Liu, Zhifang Sui - 2023

3 papers in library cite

[13]Cross-Task Generalization via Natural Language Crowdsourcing Instructions

Swaroop Mishra, Daniel Khashabi, Chitta Baral, Hananneh Hajishirzi - 2021

7 papers in library cite

[14]Llama 2: Open Foundation and Fine-Tuned Chat Models

Hugo Touvron, L. Martin, K. Stone, P. Albert, Amjad Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, Mark Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, Jaehoon Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Zhicheng Yan, I. Zarov, Y. Z. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, T. Scialom - 2023

8 papers in library cite

[15]G-Eval: NLG evaluation Using GPT-4 With Better Human Alignment

Yibo Liu, D. Iter, Yiheng Xu, Shijie Wang, Runxin Xu, C. Zhu - 2023

2 papers in library cite

[16]Are Large Language Models Good Evaluators for Abstractive Summarization?

C. Shen, L. C. Cheng, Y. You, L. Bing - 2023

1 paper in library cites

[17]Palm 2 Technical Report

R. Anil, A. M. Dai, O. Firat, M. J. Johnson, D. Lepikhin, A. Passos, Siamak Shakeri, E. Taropa, P. Bailey, Ziru Chen - 2023

3 papers in library cite

[18]Instruction Tuning With GPT-4

B. Peng, Chun-Liang Li, Pengcheng He, M. Galley, Jianfeng Gao - 2023

2 papers in library cite

[19]Stanford Alpaca: An Instruction-Following Llama model

R. Taori, I. Gulrajani, Tong Zhang, Y. Dubois, Xiang Lisa Li, C. Guestrin, Percy Liang, T. B. Hashimoto - 2023

2 papers in library cite

[20]Adapting Language Models for Zero-Shot Learning by Meta-Tuning on Dataset and Prompt Collections

R. Zhong, K. Lee, Zhengyou Zhang, Dan Klein - 2021

1 paper in library cites

[21]Automated Evaluation of Written Discourse Coherence Using GPT-4

B. Naismith, P. Mulcaire, J. Burstein - 2023

1 paper in library cites

[22]Evaluating Large Language Models on Controlled Generation Tasks

Jian Sun, Yuandong Tian, W. Zhou, N. Xu, Q. Hu, R. Gupta, J. F. Wieting, Nanyun Peng, Xueguang Ma - 2023

1 paper in library cites

[23]Gpt-4 Passes the Bar Exam

D. M. Katz, M. J. Bommarito, S. Gao, P. Arredondo - 2023

1 paper in library cites

[24]GPT-4 vs. gpt-3.5: A Concise Showdown

A. Koubaa - 2023

1 paper in library cites

[25]Gptscore: Evaluate as You Desire

J. Fu, See-Kiong Ng, Zhejun Jiang, P. Liu - 2023

1 paper in library cites

[26]Large Language Models Are Diverse Role-Players for Summarization Evaluation

N. Wu, M. Gong, L. Shou, S. Liang, D. Jiang - 2023

1 paper in library cites

[27]Towards Better Evaluation of Instruction-Following: A Case-Study in Summarization

O. Skopek, R. Aralikatte, S. Gooding, V. Carbune - 2023

1 paper in library cites

Cited by

3

papers in your library

Cites

15

papers in your library

Read

on June 6, 2026

Simple tasks, and it seems that it came out already saturated (hovering about ~80%)

Tags

Vetto StudyBenchmark

Paper Aliases

No aliases