VESTA: A Fully Automated Scenario Generation and Safety Evaluation Framework for LLM Agents

A new automated framework called VESTA has been introduced to evaluate the safety risks of large language model (LLM) agents, revealing that current systems face substantial behavioral safety risks during task execution [1]. The framework, detailed in a paper submitted to arXiv on June 7, 2026, generates 1,072 measurable evaluation scenarios based on five risk dimensions [1][2]. It then uses an automated pipeline to assess 12 LLM agents under two different authority contexts [1][2]. The results show an average attack success rate (ASR) of 47.1%, with several models exceeding 70% [1][2]. LLM agents represent an evolution beyond simple chatbots. They can maintain memory, use tools, access external environments, and execute tasks autonomously [2]. As their capabilities and autonomy expand, the diversity of safety risks they face also increases [2]. The paper argues that existing evaluation methods, which often rely on manually written scenarios, static prompts, or judgments based only on final outputs, struggle to capture the risks that emerge during the process of task execution [2]. VESTA was designed to address this gap by instantiating abstract safety risks into concrete, measurable scenarios grounded in real-world task execution [2]. The framework's fully automated nature allows for scalable, process-level evaluation rather than a simple check of an agent's final answer [2]. The findings underscore the importance of this approach for understanding and improving LLM agent safety, as the high ASR indicates that behavioral risks are not merely theoretical but are readily triggered during operation [1][2]. The paper was released through arXivLabs, a framework that allows collaborators to develop and share new features on the arXiv platform [1]. The research contributes to a growing body of work focused on the safety and alignment of increasingly autonomous AI systems.