Papperoni

2024

Chatbot Arena: An Open Platform for Evaluating LLMS by Human Preference

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tao Li, Dustin Li, Banghua Zhu, Haowei Zhang, M. Jordan, Joseph E. Gonzalez

Open PDF Google Scholar

citations

Cite Score

AI summary

This paper introduces Chatbot Arena, an open crowdsourced platform for evaluating LLMs by human preference using a pairwise comparison approach and statistical methods, amassing over 240K votes and confirming diverse, discriminating questions and agreement with expert ratings, establishing a robust, highly referenced benchmark.

Main Contributions

Introduction of Chatbot Arena, the first large-scale crowd-sourced live LLM evaluation platform utilizing human interaction for assessing LLM alignment with human preferences.
Development of an efficient model sampling and ranking algorithm based on the Bradley-Terry model and E-values, designed to accelerate ranking convergence and detect abnormalities.
Conducting an in-depth analysis of collected data, demonstrating that user-generated questions are diverse and challenging, and crowd-sourced votes are highly consistent with expert evaluations.
Public release of a human preference dataset containing over 100K pairwise votes collected from Chatbot Arena.
The platform has become a widely cited LLM leaderboard and a benchmark for the industry, making its data and code open-source and accessible.

Abstract

Large Language Models (LLMs) have unlocked new capabilities and applications; however, evalu-ating the alignment with human preferences still poses significant challenges. To address this is-sue, we introduce Chatbot Arena, an open plat-form for evaluating LLMs based on human pref-erences. Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing. The platform has been operational for several months, amassing over 240K votes. This paper describes the platform, analyzes the data we have collected so far, and explains the tried-and-true statistical methods we are using for efficient and accurate evaluation and ranking of models. We confirm that the crowdsourced questions are sufficiently diverse and discriminating and that the crowd-sourced human votes are in good agreement with those of expert raters. These analyses collectively establish a robust foundation for the credibility of Chatbot Arena. Because of its unique value and openness, Chatbot Arena has emerged as one of the most referenced LLM leaderboards, widely cited by leading LLM developers and companies. Our demo is publicly available at https://chat.lmsys.org.

Citation Graph

Loading graph...

References [51]

Sort:

Filter:

[1]Training Language Models to Follow Instructions With Human Feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, C. Wainwright, Pamela Mishkin, Chiyuan Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, Ryan Lowe - 2022

15 papers in library cite