SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents

Researchers have released SafeClawBench, a staged benchmark designed to evaluate the security of tool-using language-model agents by separately measuring semantic attack acceptance, audit-visible harm evidence, and sandbox-observed tool or state harm [1][2]. The benchmark, introduced in a paper posted to the arXiv preprint repository on June 16, 2026, includes 600 controlled adversarial tasks across six attack families: direct and indirect prompt injection, tool-return injection, memory poisoning, memory extraction, and ambiguity-driven unsafe inference [1][2]. arXiv, which began in 1991, is an open-access repository that hosts electronic preprints across disciplines including computer science, and it now receives roughly 24,000 submissions per month [6]. SafeClawBench reports three distinct endpoints. The first, semantic attack acceptance, captures whether a model verbally complies with an adversarial instruction. The second endpoint records audit-visible harm evidence, and the third logs sandbox-observed tool or state harm, such as writing to persistent memory or triggering harmful code effects [2]. The authors argue that existing evaluations often collapse these stages into a single attack success rate, obscuring whether a model merely agreed with an attacker or actually produced observable damage [1][2]. Five agent endpoints were evaluated under four prompt-level policies. Without additional prompt protection, semantic failure rates ranged from 9.0% to 44.2% across the tested models [1][2]. Audited harm evidence proved narrower than semantic failure. In a matched analysis of 12,000 rows, 291 of 347 observed sandbox harms occurred in rows that passed the semantic check, meaning the model did not verbally comply yet still produced harmful tool-level effects [1][2]. Prompt policies altered endpoint outcomes, but the effects depended on both the model and the protocol used [1][2]. The researchers note that SafeClawBench provides a reproducible framework for comparing agent models and prompt-policy conditions without conflating textual compliance, evidence-supported harm, and executable state changes [1][2]. The open-source dataset is available on Hugging Face [2]. The paper appears on arXiv with a link to the arXivLabs framework, a community collaboration space that allows third-party developers to build experimental tools on top of the repository’s article pages [4][5]. arXivLabs projects, which include citation explorers and code-finding tools, operate under guidelines that require partners to share arXiv’s values of openness, community, excellence, and user data privacy [4].