HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

A new benchmark called HOLMES reveals that large language models perform poorly on higher-order logical reasoning tasks, with the best model achieving only 59.54% accuracy across 1,379 test instances drawn from law and finance domains [1][2]. The benchmark, formally introduced in a paper by Yucheng Wu and collaborators, is described as the first real-world dataset targeting higher-order symbolic reasoning in LLMs [1][2]. Unlike existing evaluations that focus on first-order logic — deduction over fixed predicates — HOLMES requires models to reason over rules, predicates, functions, constraints, and decision procedures themselves [2]. Each of the 1,379 instances pairs a natural-language problem with a higher-order logic formalization, a ground-truth answer, and a verifiable reasoning trace [1][2]. Current LLMs averaged just 50.64% accuracy on the benchmark, while the top-performing model reached 59.54% [1][2]. The paper’s analysis further indicates that high final-answer scores can conceal shortcut reasoning when models face conflict-resolution settings, and performance drops sharply under scope-conditioned and compositional reasoning demands [2]. These results position higher-order symbolic reasoning as a bottleneck for building reliable and verifiable language models [1][2]. The work arrives as the broader AI community continues to scrutinize the reasoning capabilities of large language models, which are typically trained with self-supervised learning on vast text corpora [11]. The HOLMES dataset and project code have been made publicly available on GitHub [1][2]. The paper was posted on arXiv, an open-access repository that hosts e-prints across computer science, mathematics, and other fields and that, as of late 2024, receives roughly 24,000 new submissions per month [9].