ThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs

A new framework called ThinkProbe maps the reasoning traces of large language models into structured graphs, revealing that how a model thinks is a stable, model-level property that varies more between models than between different question domains, according to research submitted to arXiv on June 27, 2026 [1]. ThinkProbe, developed by Mohamed Amine Kerkouri and colleagues, converts each reasoning trace into a Thought Graph — a directed graph with cycles, 8 node types, and 6 edge types — and derives a 19-metric five-dimensional cognitive profile, or 5D-CP, spanning Breadth, Depth, Structure, Metacognitive, and Efficiency [1][3]. The pipeline is fully non-generative, combining rule-based segmentation and discriminative semantic linking, which the authors argue eliminates the circularity of using one LLM to analyze another [3]. The study applied ThinkProbe to 4,200 traces from 7 native reasoning models across 200 open-ended questions and 10 cognitive domains [1][3]. The analysis found that between-model variance exceeded between-domain variance by up to fourfold across four of the five cognitive dimensions [1][3]. Only the Structure dimension showed genuine sensitivity to the question domain, exposing qualitatively distinct cognitive profiles that accuracy-based evaluation cannot detect [1][3]. This work arrives amid broader efforts to move beyond final-answer accuracy and token-count metrics for evaluating reasoning models. A separate study introduced a pipeline that converts unstructured traces into verifiable reasoning graphs of claims and dependencies, finding that structural measurements can separate behaviors that token count and accuracy conflate [4]. That work also defined a reasoning-flow efficiency metric, η, which measures how concentrated a model’s logical flow is between observations and a proposed solution, differentiating traces even when they reach the same correct answer with similar length [4]. Another recent framework, TRACE, decomposes reasoning into minimally complete sub-thoughts and constructs granular thought progression graphs to identify common thinking patterns [5]. That analysis found that long-thinking models can be five to twenty times slower on simple tasks with no substantial accuracy gains, driven primarily by over-verification and over-exploration [5]. Based on those structural insights, the authors proposed a utility-based redefinition of overthinking that moves beyond length-based metrics [5]. Similarly, the ReasoningFlow framework captures discourse structures of reasoning traces into fine-grained directed acyclic graphs with 8 node types and 14 edge types, applied to 1,260 traces spanning math, science, and argumentation tasks [8]. That study found that large reasoning models exhibit structurally similar traces despite being trained from different base models and potentially non-overlapping post-training data [8]. ThinkProbe’s authors note that all 19 metrics in the 5D-CP significantly discriminate between models, establishing reasoning structure as a predominantly stable model-level property invisible to accuracy-based evaluation [3]. Future work includes correlating 5D profiles with downstream task performance to enable real-time structural confidence signals during inference, expanding to instruct models to probe the boundary between elicited and native chain-of-thought, and applying ThinkProbe to open-ended deployment contexts such as moral reasoning, creative ideation, and strategic planning, where accuracy-based evaluation is unavailable [3].