acceptodds
Under review as a conference paper at ICLR 2027

Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models

Abstract

Reinforcement learning with binary correctness rewards trains correctness, not calibrated confidence. The confidence that reasoning models verbalize is systematically overconfident, and the problem is not merely one of scale: verbalized confidence tracks how willing a model is to commit to an answer, not how likely the answer is to be right. Post-hoc rescaling therefore fits one distribution but rarely transfers. We look inside the model instead. On factual question answering, a linear probe on the hidden state between the chain of thought and the answer is substantially better calibrated: its expected calibration error is 5–38 lower than that of the verbalized score across four benchmarks and two model families. However, when used to pick among sampled answers, that same probe nearly ties majority voting yet falls far short of the oracle. Internal states answer "how certain am I” well and "which answer is right” poorly, so the signal should be reported as a confidence rather than used to select answers. As a result, we introduce probe-guided self-distillation (Probe-SD): score a model's own sampled traces with the probe, overwrite the confidence each trace states, and finetune the base checkpoint of the same family, so nothing but the model itself remains at test time. On Qwen3-14B, Probe-SD cuts ECE from 0.178 to 0.024 in-domain and from 0.542 to 0.113 out-of-domain, where it also beats post-hoc recalibration and self-consistency distillation. The resulting confidence is well-calibrated and useful for weighted voting—behaviors previously attributed to online RL, here obtained with supervised finetuning alone. Code will be released upon publication.

Primary area:
interpretability and explainable AI
Keywords:
Reasoning Models, Confidence Calibration, Verbalized Confidence, Hidden-State Probing, Self-Distillation, Uncertainty Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_43QgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the calibration of verbalized confidence in reasoning models. The authors first show that linear probes trained on hidden states provide better-calibrated confidence estimates than zero-shot verbalized confidence. They then propose Probe-SD, which uses these probe predictions to relabel confidence scores in self-generated trajectories and finetunes a base model on the resulting data. The method improves calibration on several factual QA benchmarks compared to popular baselines such as verbalized confidence, post-hoc recalibration, and self-consistency-based distillation.

Critical strengths
  • The observation that hidden-state probes produce significantly better-calibrated confidence estimates than verbalized confidence (for models where calibration has not been trained using RL) is interesting, although not totally surprising.

  • Results on factual QA are strong, showing significant improvements over the baselines used.

Critical weaknesses
  • Writing and presentation: The writing and presentation are extremely poor and do not meet the standard expected for publication. Important concepts are not introduced clearly, the writing is often informal, and the overall narrative is difficult to follow. A lot of the results sections simply repeat numerical values from tables without providing additional insights. This makes the paper extremely difficult to read through.

  • Missing and weak baselines: Several important baselines are missing. These include existing supervised calibration methods such as ConfTuner [1], RL-based calibration methods [2,3], and stronger classifier-based approaches. The authors acknowledge the absence of RL comparisons, but this does not justify their omission. Without these comparisons, it is difficult to assess the actual benefits of the proposed method over existing approaches.

  • Limited evaluation domains: All experiments are conducted on relatively narrow factual QA tasks. It is unclear how well the proposed approach would generalize to more complex reasoning domains such as mathematics, coding, and STEM problems. Given that the paper specifically targets reasoning models, this is a significant limitation.

  • Biased evaluation: The probe hyperparameters are selected using validation data from EntityQuestions and NQ-open, two of the three datasets subsequently reported as out-of-domain. Although the test sets are held out, this is not a clean evaluation of generalization to unseen domains.

  • Limited novelty: It is already well established that hidden-state probes contain information predictive of answer correctness [4]. The main contribution of this work is therefore distilling this signal into verbalized confidence, which is relatively narrow. There is already substantial prior work on training verbalized confidence using RL and SFT with empirical-accuracy-based supervision.

  • Motivation for answer-selection experiments: I do not understand the motivation for the paper's emphasis on demonstrating that the probe is a poor answer selector. The main contribution is distilling probe-based confidence into verbalized confidence, and it is unclear how answer-selection contributes to the story.

[1]: Li, Y., Xiong, M., Wu, J., & Hooi, B. (2026). Conftuner: Training large language models to express their confidence verbally. Advances in Neural Information Processing Systems, 38, 53484-53513.

[2]: Damani, M., Puri, I., Slocum, S., Shenfeld, I., Choshen, L., Kim, Y., & Andreas, J. (2026, April). Beyond binary rewards: Training lms to reason about their uncertainty. In International conference on learning representations (Vol. 2026, pp. 59456-59493).

[3]: Bani-Harouni, D., Pellegrini, C., Stangel, P., Özsoy, E., Zaripova, K., Navab, N., & Keicher, M. (2026, April). Rewarding doubt: A reinforcement learning approach to calibrated confidence expression of large language models. In International Conference on Learning Representations (Vol. 2026, pp. 117927-117943).

[4]: Zhang, A., Chen, Y., Pan, J., Zhao, C., Panda, A., Li, J., & He, H. (2025). Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification. arXiv preprint arXiv:2504.05419.

Questions for authors
  • Comparison to existing calibration methods: Why are important methods such as ConfTuner and RL-based calibration training omitted? How does Probe-SD compare to these approaches?

  • Classifier baselines: How does the proposed approach compare against stronger correctness classifiers trained with the same supervision, including nonlinear probes or fully trained classifier heads?

  • Answer selection: Why is demonstrating poor answer-selection performance a central focus of the paper?

  • Generalization across domains: Can the authors evaluate the proposed method on reasoning-intensive benchmarks in mathematics, coding, or STEM?

  • Novelty relative to prior work: Please clarify the novelty of the work given the existence of closely related papers (see weaknesses)

Reviewer_tD3GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the distinction between confidence calibration and candidate-answer selection in reasoning models. It trains a linear correctness probe on pre-answer hidden states and uses the resulting scores to relabel confidence in sampled reasoning traces. Probe-guided self-distillation then transfers these targets to a model that reports confidence without requiring a probe at deployment. Experiments on two model families and four factual-QA datasets compare calibration, answer selection, and several training controls. The results support improved verbal confidence, but the interpretation of unseen-domain transfer and the generality of the negative answer-selection conclusion require tighter evidence.

Critical strengths
  1. The paper addresses a useful deployment problem and distinguishes calibration from answer accuracy and answer selection. Transferring an internal confidence signal into a model’s verbal output is practically relevant when hidden states or a separate probe are unavailable at deployment. The calibration–selection distinction is also a worthwhile empirical question.

  2. The experimental study is substantial. It includes two model families, multiple QA datasets, matched SFT and relabeling controls, three SFT seeds, and missing-confidence sensitivity checks. The relabeling ablation changes calibration substantially while leaving accuracy nearly unchanged, and the evaluation includes AUROC and Brier scores as well as ECE. These controls support a real confidence-learning effect rather than only an accuracy change.

Critical weaknesses
  1. The headline OOD comparison gives methods unequal access to target-domain validation information. Appendix C and Appendix F describe selecting the probe layer/readout/regularization using TriviaQA, EntityQ, and NQ-open validation data, while EntityQ and NQ-open are later included in the OOD evaluation. In contrast, the post-hoc calibrators are fitted only on TriviaQA and transferred unchanged (§5.1). This is not demonstrated test-set leakage, but it weakens the unseen-domain claim and confounds the comparison with ID-only calibration baselines. This protocol issue is the main reason for my current weak-reject recommendation.

  2. The claim that internal states are useful for calibration rather than answer selection exceeds the tested setting. Table 7 shows that max-score selection with this particular linear probe is close to majority voting; it does not establish a general absence of answer-selection information in internal states. Strong global discrimination can reflect differences in question difficulty without correctly ranking candidates within a question. In addition, improvement over the original model on SimpleQA should be distinguished from adequate absolute calibration. The negative conclusion should be scoped to the probe and selector actually evaluated.

Questions for authors
  1. Decision-critical protocol check: How do the OOD results change when all probe configuration choices use TriviaQA validation only? Alternatively, can you compare with calibration baselines given equivalent target-domain validation access and label that regime explicitly?

  2. Clarification: Please report SimpleQA separately from EntityQ and NQ-open, and distinguish relative calibration improvement from the absolute ECE/Brier performance on this previously unseen domain.

  3. Decision-relevant diagnostic: On questions with both correct and incorrect sampled candidates, how well does the probe rank candidates within the same question? This would help determine whether the selection result reflects a limitation of the current probe/readout or a broader calibration–selection distinction.

Reviewer_iBL1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper show that hidden-state probing provides a better confidence signal than verbalized confidence (Table 1), but this signal does not translate into significant improvements in best-of-N selection (Figure 2). To address this, they propose Probe-guided self-distillation (Probe-SD), which trains the model to generate verbalized confidence scores using probe scores as targets. Across two model families, Probe-SD achieves better calibration on both in-distribution and out-of-distribution distillation baselines using empirical accuracy or answer-cluster frequency as labels.

Critical strengths
  • To the best of my knowledge, distilling probe scores into verbalized confidence is a novel approach, and its higher performance over empirical question-level accuracy and semantic output frequency labels is an interesting finding.
  • They empirically compare various methods for supervising verbalized confidence.
Critical weaknesses

1. Missing references.

  • [1] This paper has already show analysis regarding verbalized confidence and probe-based confidence from a calibration perspective.
  • [2] This paper has suggested distilling empirical question-level accuracy into verbalized confidence.
  • Without citing these two papers, it is difficult to clearly establish the scope of the paper's novel contributions.

[1] https://arxiv.org/abs/2406.08391

[2] https://proceedings.mlr.press/v306/jang26a.html


2. Insufficient evidence of the effect of trace-level confidence information.

Before explaining this weakness, I want to clarify one concept, There are two types of confidence to consider:

  • Question-level confidence: pq=P(Y=1∣Q=q)p_q = P(Y=1|Q=q), which represents the probability that the model produces a correct answer to a given question. This can be closely related to question difficulty.
  • Reasoning trace confidence: pg,h=P(Y=1∣Q=q,H=h)p_{g,h} = P(Y=1|Q=q, H=h), where HH denotes the hidden-state representation of the reasoning trace. This represents the probability that the answer is correct, conditioned on the internal information from the actual reasoning trajectory.

In lines 363-364, the paper argues that Probe-SD captures the latter more effectively than EA-SD or SC-SD, thereby achieving better calibration. However, the current results are consistent with at least two possible explanations:

  • The probe primarily learns question difficulty, with limited trace-specific signal => In this case, the improvement may simply reflect the probe's ability to estimate question-level difficulty more accurately than others.
  • The probe captures useful trace-specific information, but this does not translate into substantial accuracy improvements under max-probe selection.

Overall, the current results demonstrate the effectiveness of probe- supervision, but they do not sufficiently show that the improvement specifically comes from trace-specific confidence information.


3. Limited scope of ood generalization.

In line 317, the paper emphasizes robustness to ood shifts. However, current ood evaluation is largely limited to shift within factual question answering. Therefore, the current results do not necessarily establish task-general confidence calibration. To support such a claim, the method should also demonstrate generalization to different task, such as MATH or AIME.

Furthermore, if the probe primarily learns shortcuts related to question-level difficulty, these signals may transfer across factual QA datasets, since they share similar sources of uncertainty. However, mathematical reasoning tasks may involve fundamentally different sources of uncertainty, making such transfer less likely.

Therefore, without additional evaluations on tasks beyond factual QA, the paper should narrow its generalization claims to confidence calibration within the factual QA domain, rather than suggesting broader generalization.

Questions for authors

1. Within-question auroc analysis.

Could the authors report the within-question auroc of the hidden-state probe, restricted to questions with both correct and incorrect trajectories?

The overall auroc in Table 1 may primarily reflect question-level difficulty rather than trace-specific correctness. This is particularly relevant given Figure 2, where max-probe performs comparably to majority voting despite its high auroc. A within-question analysis would help distinguish these explanations.

2. Question-mean Probe-SD ablation.

Could the authors compare Probe-SD against a question-mean Probe-SD baseline, where all trajectories for the same question are supervised using their average probe score?

pˉq=1N∑i=1Npprobe(q,ti)\bar p_q = \frac{1}{N}\sum_{i=1}^{N}p_{\mathrm{probe}}(q,t_i)

Unlike EA-SD and SC-SD, this baseline would preserve the source of supervision while removing trace-specific variation. This would help determine whether Probe-SD's gains come from trace-level information or better question-level confidence estimation.

3. Cross-task generalization beyond factual QA.

Could the authors evaluate the frozen probe and Probe-SD student on mathematical reasoning benchmarks such as MATH or AIME, without additional fine-tuning or recalibration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.