SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

A new benchmark called SEVRA-BENCH tests whether large language models acting as automated code reviewers can be tricked into approving malicious pull requests, revealing a sharp security gap between closed- and open-source models [1]. SEVRA-BENCH, introduced in a paper submitted to arXiv on 11 June 2026, contains 1,062 malicious pull requests constructed from real-world vulnerabilities [1]. Each pull request is built by taking a commit that previously fixed a vulnerability listed in the Common Vulnerabilities and Exposures database, inverting the fix to restore the vulnerable code, and wrapping the submission in one of 15 social-engineering framings [1]. These framings vary the claims made, the supporting evidence, the urgency conveyed, signals of prior approval, and appeals to authority [1]. The vulnerabilities are drawn from the top 10 entries of the 2025 Common Weakness Enumeration Top 25 [1]. The benchmark addresses a question that existing static vulnerability detection and code generation benchmarks do not: whether an automated reviewer can reject a malicious contribution when the attacker controls both the code change and the accompanying pull-request text [1]. The researchers evaluated eight current large language models as code review agents in a realistic setting, using pull requests that introduce vulnerabilities previously reported in public disclosures [1]. The results showed a sharp gap in security capabilities between closed- and open-source models [1]. Large language model reviewers are increasingly used in pull-request workflows, where their approvals help decide which code is merged into a repository [1]. The growing reliance on automated review agents has raised concerns about adversarial submissions that exploit the reviewer's inability to distinguish between legitimate and malicious changes when the accompanying text is crafted to deceive [1]. The benchmark's authors hope SEVRA-BENCH will serve as a resource for advancing open-source models and narrowing the security gap [1]. The paper was submitted through arXivLabs, a framework that allows collaborators to develop and share new arXiv features directly on the website [1]. arXiv states that both individuals and organizations working with arXivLabs have embraced values of openness, community, excellence, and user data privacy [1]. The SEVRA-BENCH paper is also linked on the Hugging Face Hub, where paper pages allow the community to find related models, datasets, and demos, and to discuss the research [4]. Hugging Face and arXiv have collaborated to embed demos directly alongside papers on arXiv abstract pages, enabling users to try out open-source implementations without writing code [5]. Open-source large language models such as those from DeepSeek and Qwen have gained attention for their cost-effective training and permissive licensing [7][9]. DeepSeek, a Chinese AI company founded in July 2023, makes its models available under free and open-source software licenses, primarily the MIT License [7]. Qwen, developed by Alibaba Cloud, distributes many of its models under the Apache 2.0 license or other open-source terms [9]. The SEVRA-BENCH findings suggest that such open-source models currently lag behind their closed-source counterparts in security-focused code review tasks [1].