Process-Verified Reinforcement Learning for Theorem Proving via Lean

A new reinforcement-learning method for automated theorem proving uses the Lean proof assistant as a process-level oracle, supplying dense, structured feedback instead of a single binary reward, according to research published on arXiv [1]. The work addresses a gap between the rich, step-by-step structure of formal proofs and the sparse outcome signals typically used in reinforcement learning from verifiable rewards (RLVR) [1]. The authors parse proof attempts into tactic sequences and rely on Lean's elaboration to mark both locally sound steps and the earliest failing step, producing credit signals grounded in type theory [2]. These signals are incorporated into a GRPO-style objective that balances outcome- and process-level advantages through first-error propagation and first-token credit methods [2]. Experiments with two models, STP-Lean and DeepSeek-Prover-V1.5, showed that tactic-level supervision outperformed outcome-only baselines in most settings [2]. Improvements were recorded on formal-mathematics benchmarks MiniF2F and ProofNet [2]. DeepSeek-Prover-V1.5 is a product of Hangzhou-based DeepSeek, a Chinese artificial-intelligence company founded in July 2023 and funded by the hedge fund High-Flyer [6]. The company has drawn attention for training large language models at costs reported to be far lower than those of U.S. competitors, with its V3 model trained for roughly US$6 million compared to an estimated US$100 million for OpenAI's GPT-4 [6]. Benchmarks such as MiniF2F and ProofNet are standardized tests that provide datasets and evaluation metrics to compare model capabilities in areas including reasoning and generation [4]. The new approach's gains on these benchmarks suggest that process-level feedback can sharpen a model's theorem-proving ability beyond what binary outcome rewards alone can achieve [2]. The study frames symbolic proof assistants not merely as evaluators at test time but as training-time reward oracles [2]. This perspective connects to broader concerns about reproducibility in machine-learning research, where failures to replicate published results have prompted the growth of metascience as a discipline [5]. By rooting rewards in a deterministic, verifiable symbolic process, the method may reduce the variability that often complicates replication efforts [2][5].