Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty

A new framework proposes measuring the consistency of logical reasoning in large language models by analyzing how stably a model ranks its own outputs, rather than relying solely on whether its final answers vary. The approach, detailed in a paper submitted to arXiv on June 15, 2026, introduces a metric called structural uncertainty [1]. The work targets a known limitation: large language models can reach the same conclusion through contradictory or unstable reasoning paths, a failure mode common in multi-step deductive tasks [1]. Current reliability assessments typically focus on output dispersion, or how much sampled answers differ, which the authors argue discards a complementary signal about the model's internal consistency [1]. The framework operates by generating multiple candidate solutions for a query and then prompting the model to judge pairwise preferences among its own outputs [1]. These self-preferences are aggregated into ranking distributions using Bradley-Terry modeling with PageRank [1]. The resulting signal is decomposed into two entropy-based components: across-trial ranking instability and within-trial candidate ambiguity [1]. The researchers tested the method across five large language models and eight benchmarks [1]. They found that the structural signals provided information complementary to answer dispersion. On logical and mathematical reasoning tasks, combining the two improved the identification of unreliable instances [1]. On factual retrieval tasks, however, the structural signal collapsed toward uniformity, which the authors interpret as a regime boundary where reasoning-level consistency evaluation becomes uninformative [1]. The two components of structural uncertainty relate differently to accuracy. Within-trial ambiguity showed a positive correlation with correctness, consistent with settings where multiple plausible solution paths remain competitive [1]. Across-trial instability correlated negatively, signaling unreliable reasoning [1]. The paper frames structural uncertainty not as a universal confidence estimator, but as a regime-sensitive evaluator of logical reasoning consistency [1]. The submission, a single file of 8,497 KB, was posted by Baishali Chaudhury [1]. The paper appears on arXiv, an open-access repository for electronic preprints that, as of November 2024, receives about 24,000 articles per month [6]. The work is accessible through the arXivLabs framework, a community innovation space launched in 2020 to allow collaborators to develop and share experimental tools directly on the site [5]. arXivLabs projects, which include bibliographic explorers and code finders, operate under guidelines that require partners to share arXiv’s values of openness, community, excellence, and user data privacy [4][5].