Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

Advances in coding agents are transforming software engineering, but current benchmarks may be misaligned with this new paradigm, according to recent research^[1].

Coding agents have become a major mode of software engineering, with a new study arguing that existing benchmarks were designed in a pre-agent era^[1]. A coding agent is not just a model, but a system harness comprising models, harnesses, contexts, environments, and feedback signals. The current benchmarking approach collapses these components into a single end-to-end score, typically computed against one reference solution. This methodology has several limitations, including conflating the model with the rest of the harness, penalizing equally valid alternative solutions, and lacking signal at the level of individual harness components. Another study examines the concept of 'vibe coding,' which uses natural language prompts to build applications without requiring knowledge of code syntax^[2]. This approach promises to enable programming in a user's mother tongue, rather than relying on code syntax. The research aims to evaluate the viability of vibe coding for greenfield software engineering tasks and analyze the benchmarks used to measure its software engineering capabilities.