- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on confidence calibration of LLMs. In particular, it introduces an alternating approach to optimize two components: (1) the confidence estimator, which regresses the bootstrapped confidence from the model; and (2) the model itself, which externalizes the confidence in its text output. The authors empirically show that the proper training schedule of the confidence estimator is key to its success in providing accurate rewards. Furthermore, training the two components alternately is better than training them jointly.
Critical strengths
- The paper method is simple to implement and does not introduce too much overhead to the model itself.
- The paper provides comprehensive ablation study, covering different aspects.
Critical weaknesses
-
The novelty of this paper is too weak. The paper basically glues together two precedents: the estimator-based and verbalized approaches. Each line is already well explored, including the confidence estimator, the reward design for the verbalized score, etc.
-
The paper’s core motivation is also very weak. Specifically, the authors show in Section 2.2 that estimator-based approaches’ underperformance is due to overfitting, and the fix is essentially early stopping, which is very very basic practice. The paper never shows that earlier baselines were actually overfit. So this finding is a useful corrective rather than a contribution in its own right. Reading the Introduction, I also found the flow abrupt. The authors first show that estimator-based approaches can outperform verbalized approaches with proper training. Then they suddenly say, “let’s combine the merits of the two kinds of approaches.” This framing sounds very shallow and does not truly identify the essential research problem or motivate the solution.
-
The presentation flow in Section 2.2 and Figure 1b is horrible. Figures 2 and Figure 4 are not of publication quality.
Questions for authors
- The work is very similar to https://arxiv.org/abs/2604.23333. Discussion and comparison?
- Comparison with RewardDoubt? (In addition, RewardDoubt is accepted to ICLR 2026, so the reference source needs to be updated).
6 comments
Question 1. Comparison with RLCM.
Both RLCM and IPoET use hidden-state confidence as feedback for policy training. Their main differences lie in how the estimator is optimized and coupled to the policy, and how confidence supervision and rewards are constructed:
- Iterative training design. RLCM updates a lightweight probe online using the evolving policy’s hidden states. IPoET uses a separate, fully fine-tuned estimator and alternates policy optimization with estimator refinement. The entire estimator remains frozen throughout each policy-update phase, providing a fixed scoring function, and is subsequently refitted on fresh rollouts to accommodate the updated response distribution. Our controlled comparison in Table 2 shows that this alternating design improves confidence estimation over joint training within our framework.
- Supervision and reward. RLCM supervises its probe using Monte Carlo correctness estimates from forced completions of intermediate reasoning prefixes and constructs a confidence-margin reward. IPoET directly supervises its estimator using the verified correctness of complete responses and uses the frozen estimator’s Brier error as policy feedback. Although RLCM’s Final-Brier ablation also uses a Brier penalty, it retains the same online probe-based estimator and does not implement IPoET’s independent estimator or alternating training procedure.
Beyond these methodological differences, our empirical analyses further examine the training considerations behind this design. Section 2.2 shows that increasing estimator training budgets can lower training error while degrading held-out confidence estimation, highlighting the importance of controlling estimator optimization. Figure 3 shows that policy optimization improves confidence estimation with the estimator held fixed, followed by further gains from estimator refinement. Together, these findings support treating estimator fitting and reasoning optimization as interdependent stages in IPoET.
Additional empirical comparison. We reproduce RLCM using its official implementation, retaining its confidence-margin reward and Monte Carlo sampling mechanism. We use Qwen3-8B-Base as the backbone, with the same HotpotQA training data, common policy-training settings, and evaluation protocol as our main experiments. The results are summarized below.
| Evaluation | Method | Acc. (%) ↑ | AUROC ↑ | Brier ↓ | ECE ↓ |
|---|---|---|---|---|---|
| HotpotQA | RLCM | 63.1 | 0.585 | 0.268 | 0.174 |
| HotpotQA | IPoET (ours) | 63.3 | 0.800 | 0.184 | 0.107 |
| OOD average | RLCM | 62.9 | 0.590 | 0.275 | 0.251 |
| OOD average | IPoET (ours) | 62.3 | 0.738 | 0.170 | 0.146 |
IPoET improves AUROC by 0.215 on HotpotQA and 0.148 averaged over the six OOD datasets, with lower Brier scores and ECE in both settings. Consistent with our internal ablations, these results support the effectiveness of iterative policy–estimator training in improving confidence discrimination and calibration.
We will cite and discuss RLCM, incorporating these methodological distinctions, comparative results, and reproduction details into the manuscript.
Question 2. Comparison with Rewarding Doubt.
Rewarding Doubt uses reinforcement learning with a logarithmic scoring reward to calibrate the confidence verbalized for a generated answer [3]. IPoET instead uses an independent estimator’s confidence feedback to optimize reasoning and answer generation, with an iterative training design that alternates policy optimization and estimator refinement.
Additional comparison. We reproduce Rewarding Doubt using its official code, retaining its PPO training framework and reward implementation. We use Qwen3-8B-Base with the same HotpotQA training data and evaluation protocol as our main experiments. Results on HotpotQA and the average across six OOD datasets are shown below.
| Evaluation | Method | Acc. (%) ↑ | AUROC ↑ | Brier ↓ | ECE ↓ |
|---|---|---|---|---|---|
| HotpotQA | Rewarding Doubt | 53.0 | 0.728 | 0.223 | 0.129 |
| HotpotQA | IPoET | 63.3 | 0.800 | 0.184 | 0.107 |
| OOD average | Rewarding Doubt | 53.8 | 0.733 | 0.211 | 0.202 |
| OOD average | IPoET | 62.3 | 0.738 | 0.170 | 0.146 |
IPoET achieves higher accuracy and AUROC, together with lower Brier scores and ECE, in both settings. Its HotpotQA AUROC improves by 0.072, while OOD Brier and ECE decrease by 0.041 and 0.056, respectively. These results support the effectiveness of our iterative policy–estimator training approach for improving confidence estimation alongside task performance.
We will incorporate this comparison and its reproduction details into the manuscript and update the reference to the published ICLR 2026 version.
[3] Bani-Harouni et al. Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models. ICLR 2026.
Thank you for recognizing the simplicity of IPoET, its practical efficiency, and the comprehensive ablation study. Before addressing the concerns below, we would like to clarify two potential misunderstandings in the summary. First, the estimator is supervised by verified answer correctness, rather than bootstrapped confidence targets. Second, the policy generates only the reasoning trace and final answer. The estimator then processes the question and response to produce the numerical confidence score. This design allows confidence estimation to benefit from reasoning without requiring the policy to generate additional analysis of its own confidence. We would appreciate your reconsideration of the novelty and significance of our work in light of these clarifications and the evidence presented below.
Weakness1. Novelty.
Beyond our empirical study, the methodological novelty of IPoET lies in its iterative training design, which integrates the complementary strengths of reasoning generation and hidden-state confidence estimation. A key element of this design is to construct a Brier-based policy reward from a frozen estimator’s predictions on complete responses and verified answer correctness. This provides a fixed scoring function during each policy-update phase, while subsequent estimator refinement on fresh rollouts enables adaptation to the evolving policy. The ablations in Fig. 3 and Table 2 support the benefits of this interaction and the alternating design.
Weakness 2. Overfitting and motivation.
Our empirical study focuses on the stability of estimator performance during training. As shown in Figure 1b, confidence estimation on unseen data can deteriorate even as training error continues to decrease. This makes comparisons with verbalized approaches sensitive to the estimator’s training budget and checkpoint selection. Our intention is to highlight this sensitivity and its implications for evaluating estimator-based methods. Regarding direct verification of earlier baselines, RLCR releases estimator checkpoints but does not provide the corresponding training code, so we cannot determine whether its estimator baseline was overfit. Nevertheless, the divergence between training and held-out performance observed in our experiments is consistent with overfitting and provides a possible explanation for the different rankings reported by RLCR. We will revise the wording to distinguish our empirical findings from this possible explanation for prior results.
Our motivation builds on the finding that a carefully trained estimator can outperform verbalized confidence. Prior work further shows that reasoning can improve confidence calibration [1] and expose uncertainty signals that an external model can recover from the reasoning trace [2]. These findings motivate us to investigate whether optimizing the policy’s reasoning can further improve the estimator’s confidence predictions. We therefore use estimator feedback to optimize the reasoning so that it better supports the estimator, making confidence-estimation quality an explicit objective of policy training. Such policy updates shift the response distribution, requiring the estimator to be refitted on fresh rollouts. This interdependence further motivates our iterative training design. To make this rationale more explicit, we will revise the wording in the manuscript.
Weakness3. Presentation.
Thank you for your feedback on the presentation. We wonder whether any display issues affected the readability of the figures during review. We would appreciate more specific feedback on Section 2.2 and the figures to help us address your concerns.
[1] Yoon et al. Reasoning Models Better Express Their Confidence. NeurIPS 2025.
[2] Podolak and Verma. Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMs. UncertaiNLP 2025.