QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

A new framework called QMFOL generates monadic first-order logic reasoning tasks with quantifiable complexity to benchmark large language models, according to a paper posted on arXiv [1]. The accompanying benchmark, QMFOLBench, contains 2,880 instances across 960 configurations [1]. Large language models have shown progress in deductive reasoning, which is important for high-stakes decision-making, but existing evaluation benchmarks lack fine-grained control over logical complexity and struggle to balance semantic diversity with logical consistency [1]. The QMFOL framework constructs formal logical structures using conjunction and disjunction patterns, allowing precise control over reasoning depth, width, label types, and distractors [1]. These structures are translated into natural language via LLMs, and logical consistency is verified through round-trip verification with an external prover [1]. QMFOLBench comprises 2,880 instances with 960 configurations spanning diverse logical and semantic dimensions [1]. Evaluations on six large reasoning models and two LLMs found that performance degrades and computational overhead increases as logical complexity rises [1]. Models performed better on tasks labeled True than on those labeled False or Unknown, and they showed sensitivity to semantic variation [1]. The paper was submitted to arXiv on June 18, 2026, in the computer science artificial intelligence category [1]. arXiv, which began on August 14, 1991, is an open-access repository of electronic preprints that are moderated but not peer-reviewed [6]. As of November 2024, the repository receives about 24,000 submissions per month and had surpassed two million articles by the end of 2021 [6]. Large language models are machine learning models with many parameters, trained with self-supervised learning on vast amounts of text for natural language processing tasks such as language generation [8]. The QMFOL framework offers a scalable approach for constructing deductive reasoning benchmarks with controllable complexity, which the authors argue enables more precise evaluation of reasoning capabilities in modern language models [1].