Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States

Linear probes widely used to claim language models learn distinct reasoning representations may instead be detecting superficial task formatting, according to a study of the Qwen3-14B model published on arXiv [1]. The research tested linear probing on three benchmarks spanning deductive, inductive, and abductive reasoning: LogiQA 2.0, ARC-Challenge, and αNLI [1]. At layer 32 of the model's 40 layers, probes achieved 100% cross-validated accuracy with well-separated geometric properties, including intrinsic dimensionalities of 20.6, 28.5, and 33.6, and convex hull contamination at or below 1.5% [1]. Large language models, which are transformer-based neural networks trained on vast text corpora, are foundational to modern chatbots and are routinely evaluated on benchmarks that attempt to measure reasoning and factual accuracy [3][6]. However, the apparent separation vanished when researchers controlled for format confounds. Residualizing source identity, option count, and response length reduced probe accuracy to chance levels [1]. Trace-anchor similarity analysis further indicated largely shared reasoning representations across tasks, with 42.5% agreement compared to a 33.3% chance baseline [1]. Causal steering experiments using 20 random controls showed no functional link between the geometric separation and actual reasoning mode, yielding a p-value of 0.286 [1]. The findings challenge the interpretation of linear probe results in mechanistic interpretability research. High-quality labeled training datasets for supervised machine learning are typically difficult and expensive to produce, which has made probing techniques an attractive shortcut for understanding model internals [5]. The authors recommend routine format deconfounding as a standard practice when using probes to investigate computational structure in language models [1].