Reading Calibrated Uncertainty from Language Model Trajectories

A new method for quantifying uncertainty in language models extracts geometric features from the trajectory of internal computations rather than relying on final output probabilities, according to research posted to arXiv [1]. The approach uses a sparse linear probe to outperform a widely used baseline, with gains tied to how miscalibrated the baseline is [2]. The maximum softmax probability (MSP) is a common default for evaluating uncertainty in structured language model generation, but it is often miscalibrated [2]. Existing probing methods typically feed raw hidden states into opaque classifiers, treating internal activations as static snapshots and ignoring the layer-by-layer path that forms a representation [2]. The researchers argue that similar final states can emerge from very different internal paths, and that the accumulation, reinforcement, or reversal of evidence across layers may reveal uncertainty that final probabilities hide [2]. To capture this dynamic, the team extracted eleven scale-invariant geometric features that trace the cumulative path of per-layer MLP updates [2]. These features are fed into a sparse linear probe, which outperforms MSP under selective abstention [2]. The performance gains scale with the degree of baseline miscalibration, reaching up to 21 AURC points [2]. Because each feature carries a closed-form geometric meaning, the probe's learned coefficients can be inspected to trace how and where errors take shape along the model's depth [2]. This analysis can identify layers that commit prematurely, layers that contradict the running state, and points where trajectories drift away from their final endpoint [2]. The work provides a more transparent alternative to opaque probing classifiers by making the trajectory of computation itself the object of study. The method does not require modifying the underlying language model and operates by post-processing its internal updates [2]. The research was submitted to arXiv on 19 May 2026 [1].