Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs

A new pipeline called Mask-Proof automatically curates and evaluates mathematical proofs using large language models, creating a benchmark of 292 problems across diverse research areas, according to a paper posted to arXiv. [1] The system addresses a persistent gap in measuring step-level reasoning within long proofs. Existing evaluations tend to focus on final answers or require expensive expert grading, while end-to-end proof generation remains difficult to verify automatically. [1] Mask-Proof converts real proofs into masked-step tasks: it hides key formula steps, supplies surrounding context, and then uses an LLM-based equivalence judge with repeated voting to assess model reconstructions. [1] The resulting dataset, Mask-ProofBench, spans multiple research domains. [1] In experiments with 17 models, reasoning-enhanced architectures outperformed standard models by 12% to 27%. [1] The automated evaluator achieved 96.8% agreement with expert annotators, which the authors argue enables faithful and reproducible measurement of step-level mathematical reasoning. [1] This work arrives as the broader AI community grapples with how to benchmark models on tasks that require deep, multi-step logic rather than pattern matching. The paper’s emphasis on automatic verification echoes earlier efforts in scientific machine learning, where researchers explored transfer learning across computational chemistry datasets to improve model performance without manual labeling. [4] Those efforts showed that models trained on one large dataset could be fine-tuned on smaller, specialized ones—a strategy that parallels the Mask-Proof team’s goal of creating a scalable evaluation framework. [4] The benchmark and associated code have been released on GitHub, and the project is also listed on Hugging Face. [1] The authors frame the pipeline as a step toward trustworthy AI assistance in proof-certified scientific progress, though they note that fully automated proof generation remains an open challenge. [1]