DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents

A new automated framework called DualGauge reveals that large language models and coding agents struggle to produce code that is simultaneously correct and secure, with even the strongest systems falling below 15% joint success across three programming languages. The framework, described in a paper posted to the arXiv preprint repository, is the first fully automated system for jointly evaluating functional correctness and security in specification-only code generation [1][2]. It is paired with DualGauge-Bench, a language-agnostic benchmark comprising 307 coding tasks, each equipped with functional and security tests derived from the same natural-language specification [2]. Researchers evaluated 10 representative LLMs across Python, C++, and JavaScript [2]. Functional correctness alone substantially overestimates reliable code generation, the authors report. Even the strongest model remained below 15% joint security-functionality success in every language tested [1][2]. Common model-side factors — including scale, extended thinking, quantization, instruction tuning, and code specialization — did not reliably improve joint performance, indicating that secure-and-correct code generation does not simply emerge from stronger coding capability [2]. The study also examined three leading agentic coding systems: Codex, OpenHands, and Claude Code [1][2]. Iterative scaffolding provided no advantage over direct LLM-based generation on specification-only tasks, the authors found [2]. A qualitative audit traced the majority of failures to the output contract boundary and to security guards that exist but are insufficient — patterns that only joint benchmarking reliably exposes [2]. arXiv, where the paper was submitted in November 2025 and revised in June 2026, serves as an open-access repository for electronic preprints across physics, mathematics, computer science, and related fields [6]. The repository passed two million articles by the end of 2021 and now receives roughly 24,000 submissions per month [6]. The DualGauge paper appears under the Software Engineering category (cs.SE) [1]. The work was led by Rupam Patir and colleagues [1]. The findings suggest that current evaluation practices, which often measure functional correctness in isolation, may paint an overly optimistic picture of LLM-generated code reliability [2].