InvestPhilBench: A Multi-Layer Dynamic Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy

A new benchmark called InvestPhilBench tests whether large language models can reconstruct and apply the procedural decision frameworks of expert investors, spanning eight cognitive tiers from principle identification to novel framework extrapolation [1][2]. The v0.6 release of InvestPhilBench, detailed in a paper posted to the arXiv preprint repository on June 24, 2026, comprises 118 primary-source-verified investment principle cards, 25 decision framework cards with explicit topology metadata, and 243 question-answer pairs split into 197 development and 46 held-out test items [1][2]. arXiv, which began on August 14, 1991, hosts preprints across mathematics, physics, computer science, and other fields and now receives about 24,000 submissions per month [6]. To enable reproducible scoring at scale, the authors introduce the Benchmark Automated Scoring Pipeline, or BASP, which uses five algorithmic metrics: OGRS, KCCS, SAP@k, IVP, and CKCA [1][2]. The pipeline also includes a Failure Mode Detection Protocol with computable rules for six failure modes, and a per-gate metric called Gate Reconstruction Accuracy for questions that have gold reasoning programs [2]. A four-model sanity wave run on the 188-question development split produced a sharp provider-tier split, with BASP composite scores of 0.906 for the top-performing model and 0.438 for the lower-performing model [1][2]. The authors note these mixed-judge numbers are confounded upper bounds [2]. The central finding is that the BASP composite saturates at the frontier — Claude L4 scored 0.932 — while Gate Reconstruction Accuracy still exposes a procedural deficit, with the frontier L4 GRA at approximately 0.77 and L7 GRA in the 0.57–0.62 range [1][2]. Composite scoring rewards fluent prose and hides the procedural gap, the paper states [2]. On a 100-item expert-annotated gold set, the automated BASP composite tracked the human reference with a Pearson correlation of 0.72 and a mean absolute error of 0.10 [1][2]. The attribution sub-metric SAP@3 was the weakest, and the failure-mode detector was described as sensitive but over-flagging [2]. The v0.6 release implements a unified judge and true model-in-the-loop retrieval and oracle conditions; the de-confounded multi-model leaderboard and full three-condition run are listed as v1.0 deliverables [2].