REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs

Researchers have introduced REALM, a unified red-teaming benchmark designed to probe vulnerabilities in vision-language models used as perception-reasoning backbones in safety-critical physical systems [1][2]. The framework integrates 12 attack methods, three model-agnostic defenses, and 13 VLMs under a shared black-box threat model to enable direct comparison of adversarial techniques [1][2]. Vision-language models are increasingly deployed in embodied intelligence applications where perception or reasoning errors can lead to unsafe actions, yet existing red-teaming benchmarks have remained fragmented across datasets, metrics, and threat models [1][2]. Chatbot-centric evaluations standardize jailbreak and content-safety testing but do not systematically capture physically grounded functional failures [1][2]. The REALM benchmark addresses this gap by introducing an agentic target-generation pipeline that constructs shared, scenario-specific attack objectives for each scene, aligning adversarial goals across attack families [1][2]. Under the unified protocol, text and typographic injection attacks induced the most failures among the tested VLMs [1][2]. The evaluation also found that multimodal co-optimization yielded the strongest visual-perturbation transfer, single-pass attacks approached the effectiveness of iterative methods at much lower cost, and model scale alone did not confer adversarial robustness [1][2]. Code for the benchmark is publicly available on GitHub [1][2]. The paper was submitted to arXiv on June 22, 2026, under the Computer Vision and Pattern Recognition category [1]. arXiv, an open-access repository of electronic preprints founded in 1991, hosts papers across mathematics, physics, computer science, and related fields, with submissions now averaging about 24,000 articles per month as of late 2024 [6]. The repository surpassed two million articles by the end of 2021 and is not peer-reviewed, though submissions undergo moderation before posting [6]. The REALM paper appears with the arXivLabs framework, a community collaboration initiative that allows third-party developers to build experimental tools on the arXiv platform [4]. Launched in 2020, arXivLabs provides a formalized structure for projects such as the Bibliographic Explorer and CORE Recommender, which help readers navigate citation trees and discover related open-access research [4][5]. arXiv has stated that collaborators under this framework must adhere to values of openness, community, excellence, and user data privacy, and are granted only minimal, anonymized data access for feature functionality [4]. As of the benchmark's publication, arXiv had temporarily paused new Labs proposals while its development team focused on modernizing infrastructure and migrating systems to the cloud [3].