CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

A new protocol called CALIBER improves how reasoning language models estimate their own confidence by supervising predictions both before and after the model generates an answer, according to research published on arXiv [1]. The method, whose name stands for Calibration Before and After Reasoning, addresses a limitation in existing systems that typically elicit confidence only once — either before the model begins its chain-of-thought or after it finishes [1]. The authors argue that confidence is state-dependent: a pre-reasoning estimate should reflect the probability of eventually solving the prompt, while a post-reasoning estimate should predict whether the specific answer just produced is correct [2]. CALIBER supervises each estimate with a target matched to its information state [1]. On the BigMathDigits benchmark, CALIBER reduced Expected Calibration Error by 52.5% compared to the strongest single-confidence baseline when tested with a 7-billion-parameter model [1]. It also achieved the best Brier score and AUROC on that dataset while remaining within 2.1 points of the top accuracy figure [1]. When scaled to a 30-billion-parameter model, the protocol again posted the best ECE on BigMathDigits and stayed competitive on Brier score and AUROC [2]. Out-of-distribution testing showed further gains. CALIBER recorded the best ECE and Brier score on GPQA and TriviaQA, two datasets that differ from the training distribution, and remained competitive on SimpleQA [1]. Ablation studies indicated that aligning the supervision target with the model’s position in the reasoning process is most valuable under distribution shift, where it consistently lowered calibration error across all out-of-distribution benchmarks [2]. The work arrives as language models are increasingly deployed in settings where users need reliable confidence scores alongside answers [1]. The researchers note that without position-aware supervision, models may conflate prompt-level and answer-level uncertainty, degrading the usefulness of the confidence signal [2].