LLM-Assisted Stance Detection in Scientific Discourse: A Test Case in Bayesian Cognitive Science

Researchers deployed large language models to classify whether scientists treat Bayesian models as literal descriptions of mental mechanisms or as abstract mathematical tools, a task that has traditionally required expert human judgment [1]. The study, led by Eyup Engin Kucuk and posted to arXiv on 14 June 2026, tackled a persistent interpretive question in cognitive science: the realism-instrumentalism divide over Bayesian models of the mind [1]. The team built a theory-driven codebook and used expert-coded reference annotations to guide a diagnostic-gated prompt-optimization search, ultimately producing a shared zero-shot prompt for three frontier LLMs — GPT-5.1, Claude Sonnet 4.6, and Gemini 3 Pro Preview [1]. The final prompt achieved a held-out combined reliability score of 0.76, calculated as the harmonic mean of an Inter-Class Correlation of 0.79 and a Cronbach’s alpha of 0.74 [1]. The prompt was then deployed on 6,858 quotes extracted from 210 articles [1]. Across the three models, quote-level agreement reached an ICC of 0.80 and an alpha of 0.76, yielding a combined agreement score of 0.78 [1]. Article-level rank stability was near-perfect, with pairwise correlations ranging from 0.96 to 0.97 [1]. These metrics suggest that LLMs can replicate expert-level coding with high consistency when the task is carefully scaffolded, though the authors caution that the framework is designed for theoretically demanding cases and does not claim to generalize to all qualitative analysis [1]. The corpus-level findings revealed that the scientific literature leans weakly realist overall, but individual articles rarely maintain a single stance [1]. Only 1.4 percent of articles fell within a single stance band, while 59.5 percent spanned four or more bands [1]. The analysis also quantified a long-held qualitative intuition: articles focused on low-level perception and motor processes scored 8.8 Realism points higher than those addressing high-level cognition, a difference that was statistically significant with a p-value below .001 and a Cohen’s d of 0.60 [1]. The work arrives as computational social science increasingly grapples with scaling expert annotation. Qualitative coding remains central to the field, but human annotation is expensive and slow, making LLM-assisted approaches attractive when validated against rigorous reliability benchmarks [2]. The submission itself, weighing 1,677 KB, includes the full codebook and prompt-optimization pipeline, underscoring the transparency required for such interpretive tasks [1].