Papperoni

2024

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu

Open PDF Google Scholar

citations

Cite Score

AI summary

This paper introduces OSWORLD, a scalable real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems, and a benchmark of 369 computer tasks; it reveals that state-of-the-art LLM/VLM-based agents struggle with GUI grounding and operational knowledge, achieving only 12.24% success compared to 72.36% for humans.

Main Contributions

Introduction of OSWORLD, a first-of-its-kind scalable, real computer environment for multimodal agents supporting task setup, execution-based evaluation, and interactive learning across various operating systems (Ubuntu, Windows, macOS).
Creation of a benchmark of 369 real-world computer tasks, including web and desktop apps, OS file I/O, and multi-app workflows, with detailed initial state setup and custom execution-based evaluation scripts.
Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWORLD, revealing significant deficiencies (best model achieves 12.24% success compared to human 72.36%).
Identification of key challenges for VLM-based agents, including struggles with GUI grounding, repetitive actions, handling unexpected application windows, and limited knowledge of basic GUI interactions and domain-specific features.
Provision of valuable insights for developing multimodal generalist agents, highlighting the need for improved VLM GUI grounding, interaction commonsense knowledge, higher-resolution support, and more robust GUI interactions.

Abstract

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWORLD, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWORLD can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWORLD, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWORLD reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWORLD provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.

Citation Graph

Loading graph...

References [67]

Sort:

Filter:

[1]SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Carlos E. Jimenez, Jihan Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, K. R. Narasimhan - 2024

6 papers in library cite