Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study

A systematic analysis of five state-of-the-art Automated Program Repair agents across 500 real-world tasks finds that current systems excel at simple fixes but falter on logic-intensive bugs, often producing patches that merely satisfy existing tests rather than addressing root causes. The study, submitted to arXiv in June 2025 and revised in May 2026, traces the full decision-making pipelines of these agents — from issue description to patch validation — and represents the first empirical examination of how such systems take actions, where they fail, and how their behavior compares to that of human developers [1][2]. Researchers found that test generation and regression test selection remain major bottlenecks. Agents frequently cannot reproduce issues or run relevant regression tests, which undermines the reliability of their proposed fixes [2]. Most agents operate with primitive tooling, such as bash scripts, and lack access to debuggers or program analyzers. This constrains their reasoning and limits patch quality [2]. The findings point to a broader engineering principle: reliability depends not only on the probability that a system will perform its intended function but also on the availability of testing and maintenance infrastructure to verify that function over time [3]. The paper argues for a shift-left approach, emphasizing early, high-quality test generation and validation to reduce spurious fixes and improve semantic correctness [2]. In traditional engineering disciplines, testing and simulation are standard practices before production deployment, and forensic engineering is applied when failures occur to determine root causes and implement fixes [4]. The study suggests that APR systems have not yet integrated these practices into their agentic workflows. The authors outline three directions for next-generation APR design: richer and more integrated tool ecosystems, diversified agentic architectures that combine complementary strengths, and benchmarks that prioritize semantic repair quality and test generation fidelity over surface-level success metrics [2]. The work was led by Ira Ceka and conducted under the arXivLabs framework, which supports experimental projects developed with community collaborators who adhere to arXiv's values of openness and user data privacy [1].