FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning

A new framework called FM-Agent automates formal verification of large codebases, using large language models to apply Hoare-style compositional reasoning and uncover hundreds of bugs in previously tested systems, according to research published on arXiv [1]. The framework, detailed in a paper by Haoran Ding and colleagues, targets a persistent bottleneck in software verification: the manual effort required to write formal specifications for individual functions [1]. Hoare logic, a foundational method for proving program correctness, decomposes systems into smaller components but has historically struggled to scale because each function demands a precise specification [2]. The problem intensifies when code is generated by LLMs, as developers often lack a deep understanding of each function's expected behavior [3]. FM-Agent addresses this by introducing a top-down paradigm that derives a function's specification from how its callers expect it to behave, capturing developer intent even when the implementation contains bugs [4]. Because that intent is typically expressed in natural language, the framework generalizes Hoare-style inference to reason about functions against natural-language specifications, bypassing the formula-only limitation of existing verifiers [5]. After reasoning, FM-Agent automatically generates test cases to confirm bug existence and explain root causes [1]. In an evaluation, the framework successfully reasoned about large-scale systems within 2 days, including codebases of up to 143,000 lines of code [1]. The systems had already undergone developer testing, yet FM-Agent identified 522 newly discovered bugs [2]. The paper states these defects could lead to serious consequences such as system crashes and incorrect execution results [3]. The work lands as LLM-assisted software development becomes more prevalent, with models capable of generating large-scale systems like compilers [4]. The research community has increasingly focused on strengthening correctness guarantees for such generated code [5]. The paper appears on arXiv, a platform that has integrated with services like Hugging Face Spaces to make machine-learning research more accessible through interactive demos [8].