LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

A new benchmark called LabOSBench tests how well multimodal AI agents can operate simulated scientific instruments through a web browser, offering a low-cost alternative to evaluating agents on physical hardware [1][2]. The benchmark, detailed in a paper submitted to arXiv on 15 June 2026, constructs 96 subtasks across eight instrument simulators [1][2]. The tasks span workflows from sample loading and alignment to parameter tuning, data acquisition, and result inspection [2]. The simulators run directly in a browser, avoiding the overhead of full operating-system virtualization while still supporting flexible task configuration and execution-based evaluation [2]. The authors argue that existing computer-use benchmarks focus on software operation in virtualized systems, but scientific instruments demand coordinated control over complex interfaces and feedback-driven parameter adjustment [2]. Testing agents on real high-precision instruments is impractical because of high cost, safety risks, limited access, and the difficulty of ensuring reproducible results [2]. LabOSBench was designed to preserve those operational challenges in a simulated environment that is both scalable and safe [2]. Researchers evaluated general-purpose vision-language models, specialized GUI agent models, and advanced agentic frameworks at both the subtask and end-to-end levels [2]. The experiments showed that while current agents can complete many structured GUI subtasks, they still struggle with feedback-driven operations and long-horizon workflow execution [1][2]. The paper does not name specific models or provide quantitative success rates in its abstract. The work appears on arXiv, an open-access repository that hosts preprints across mathematics, physics, computer science, and related fields [6]. As of November 2024, the repository was receiving about 24,000 new articles per month and had surpassed two million total articles by the end of 2021 [6]. The LabOSBench paper is listed under the Computer Science > Artificial Intelligence category [1]. The paper’s abstract page also carries links to community-built tools through arXivLabs, a framework that allows third-party collaborators to develop and share experimental features on the arXiv website [4]. arXivLabs projects have included bibliographic explorers, code-and-data linkers, and recommender systems [5]. arXiv has stated that collaborators in the program must adhere to the repository’s values of openness, community, excellence, and user data privacy [4]. The arXivLabs program is currently on a temporary hiatus for new proposals while the development team focuses on migrating arXiv’s systems to the cloud [3].