acceptodds
Under review as a conference paper at ICLR 2027

When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation

Abstract

Confidence estimation is crucial for developing trustworthy large language models (LLMs), with most methods following estimator-based or verbalization-based paradigms. While recent research increasingly focuses on improving verbalized self-reports of confidence, we challenge the prevailing view that this approach surpasses independent confidence estimators. Our empirical study shows that a dedicated confidence estimator can substantially outperform verbalized confidence, indicating that LLMs' internal representations contain richer confidence signals. Building on this finding, we propose **I**terative **Po**licy-**E**stimator **T**raining (**IPoET**), a framework that synergizes the complementary strengths of verbalized reasoning traces and informative representations. IPoET alternates policy optimization with estimator updating, integrating estimator-derived confidence feedback into policy learning and refreshing the estimator on new policy rollouts. Experiments across diverse datasets and Qwen and Llama backbones demonstrate that, by iteratively exploiting richer hidden features and adapting to the evolving policy distribution, IPoET consistently outperforms both estimator- and verbalization-based baselines in-domain and achieves superior or comparable results across all out-of-domain metrics. For more details, refer to https://anonymous.4open.science/r/ipoet.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Confidence Estimation, Reinforcement Learning for LLMs, Hidden-State Representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5ufEOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on confidence calibration of LLMs. In particular, it introduces an alternating approach to optimize two components: (1) the confidence estimator, which regresses the bootstrapped confidence from the model; and (2) the model itself, which externalizes the confidence in its text output. The authors empirically show that the proper training schedule of the confidence estimator is key to its success in providing accurate rewards. Furthermore, training the two components alternately is better than training them jointly.

Critical strengths
  1. The paper method is simple to implement and does not introduce too much overhead to the model itself.
  2. The paper provides comprehensive ablation study, covering different aspects.
Critical weaknesses
  1. The novelty of this paper is too weak. The paper basically glues together two precedents: the estimator-based and verbalized approaches. Each line is already well explored, including the confidence estimator, the reward design for the verbalized score, etc.

  2. The paper’s core motivation is also very weak. Specifically, the authors show in Section 2.2 that estimator-based approaches’ underperformance is due to overfitting, and the fix is essentially early stopping, which is very very basic practice. The paper never shows that earlier baselines were actually overfit. So this finding is a useful corrective rather than a contribution in its own right. Reading the Introduction, I also found the flow abrupt. The authors first show that estimator-based approaches can outperform verbalized approaches with proper training. Then they suddenly say, “let’s combine the merits of the two kinds of approaches.” This framing sounds very shallow and does not truly identify the essential research problem or motivate the solution.

  3. The presentation flow in Section 2.2 and Figure 1b is horrible. Figures 2 and Figure 4 are not of publication quality.

Questions for authors
  1. The work is very similar to https://arxiv.org/abs/2604.23333. Discussion and comparison?
  2. Comparison with RewardDoubt? (In addition, RewardDoubt is accepted to ICLR 2026, so the reference source needs to be updated).
6 comments

Thank you for recognizing the simplicity of IPoET, its practical efficiency, and the comprehensive ablation study. Before addressing the concerns below, we would like to clarify two potential misunderstandings in the summary. First, the estimator is supervised by verified answer correctness, rather than bootstrapped confidence targets. Second, the policy generates only the reasoning trace and final answer. The estimator then processes the question and response to produce the numerical confidence score. This design allows confidence estimation to benefit from reasoning without requiring the policy to generate additional analysis of its own confidence. We would appreciate your reconsideration of the novelty and significance of our work in light of these clarifications and the evidence presented below.

Weakness1. Novelty.

Beyond our empirical study, the methodological novelty of IPoET lies in its iterative training design, which integrates the complementary strengths of reasoning generation and hidden-state confidence estimation. A key element of this design is to construct a Brier-based policy reward from a frozen estimator’s predictions on complete responses and verified answer correctness. This provides a fixed scoring function during each policy-update phase, while subsequent estimator refinement on fresh rollouts enables adaptation to the evolving policy. The ablations in Fig. 3 and Table 2 support the benefits of this interaction and the alternating design.

Weakness 2. Overfitting and motivation.

Our empirical study focuses on the stability of estimator performance during training. As shown in Figure 1b, confidence estimation on unseen data can deteriorate even as training error continues to decrease. This makes comparisons with verbalized approaches sensitive to the estimator’s training budget and checkpoint selection. Our intention is to highlight this sensitivity and its implications for evaluating estimator-based methods. Regarding direct verification of earlier baselines, RLCR releases estimator checkpoints but does not provide the corresponding training code, so we cannot determine whether its estimator baseline was overfit. Nevertheless, the divergence between training and held-out performance observed in our experiments is consistent with overfitting and provides a possible explanation for the different rankings reported by RLCR. We will revise the wording to distinguish our empirical findings from this possible explanation for prior results.

Our motivation builds on the finding that a carefully trained estimator can outperform verbalized confidence. Prior work further shows that reasoning can improve confidence calibration [1] and expose uncertainty signals that an external model can recover from the reasoning trace [2]. These findings motivate us to investigate whether optimizing the policy’s reasoning can further improve the estimator’s confidence predictions. We therefore use estimator feedback to optimize the reasoning so that it better supports the estimator, making confidence-estimation quality an explicit objective of policy training. Such policy updates shift the response distribution, requiring the estimator to be refitted on fresh rollouts. This interdependence further motivates our iterative training design. To make this rationale more explicit, we will revise the wording in the manuscript.

Weakness3. Presentation.

Thank you for your feedback on the presentation. We wonder whether any display issues affected the readability of the figures during review. We would appreciate more specific feedback on Section 2.2 and the figures to help us address your concerns.

[1] Yoon et al. Reasoning Models Better Express Their Confidence. NeurIPS 2025.

[2] Podolak and Verma. Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMs. UncertaiNLP 2025.

Question 1. Comparison with RLCM.

Both RLCM and IPoET use hidden-state confidence as feedback for policy training. Their main differences lie in how the estimator is optimized and coupled to the policy, and how confidence supervision and rewards are constructed:

  • Iterative training design. RLCM updates a lightweight probe online using the evolving policy’s hidden states. IPoET uses a separate, fully fine-tuned estimator and alternates policy optimization with estimator refinement. The entire estimator remains frozen throughout each policy-update phase, providing a fixed scoring function, and is subsequently refitted on fresh rollouts to accommodate the updated response distribution. Our controlled comparison in Table 2 shows that this alternating design improves confidence estimation over joint training within our framework.
  • Supervision and reward. RLCM supervises its probe using Monte Carlo correctness estimates from forced completions of intermediate reasoning prefixes and constructs a confidence-margin reward. IPoET directly supervises its estimator using the verified correctness of complete responses and uses the frozen estimator’s Brier error as policy feedback. Although RLCM’s Final-Brier ablation also uses a Brier penalty, it retains the same online probe-based estimator and does not implement IPoET’s independent estimator or alternating training procedure.

Beyond these methodological differences, our empirical analyses further examine the training considerations behind this design. Section 2.2 shows that increasing estimator training budgets can lower training error while degrading held-out confidence estimation, highlighting the importance of controlling estimator optimization. Figure 3 shows that policy optimization improves confidence estimation with the estimator held fixed, followed by further gains from estimator refinement. Together, these findings support treating estimator fitting and reasoning optimization as interdependent stages in IPoET.

Additional empirical comparison. We reproduce RLCM using its official implementation, retaining its confidence-margin reward and Monte Carlo sampling mechanism. We use Qwen3-8B-Base as the backbone, with the same HotpotQA training data, common policy-training settings, and evaluation protocol as our main experiments. The results are summarized below.

EvaluationMethodAcc. (%) ↑AUROC ↑Brier ↓ECE ↓
HotpotQARLCM63.10.5850.2680.174
HotpotQAIPoET (ours)63.30.8000.1840.107
OOD averageRLCM62.90.5900.2750.251
OOD averageIPoET (ours)62.30.7380.1700.146

IPoET improves AUROC by 0.215 on HotpotQA and 0.148 averaged over the six OOD datasets, with lower Brier scores and ECE in both settings. Consistent with our internal ablations, these results support the effectiveness of iterative policy–estimator training in improving confidence discrimination and calibration.

We will cite and discuss RLCM, incorporating these methodological distinctions, comparative results, and reproduction details into the manuscript.

Question 2. Comparison with Rewarding Doubt.

Rewarding Doubt uses reinforcement learning with a logarithmic scoring reward to calibrate the confidence verbalized for a generated answer [3]. IPoET instead uses an independent estimator’s confidence feedback to optimize reasoning and answer generation, with an iterative training design that alternates policy optimization and estimator refinement.

Additional comparison. We reproduce Rewarding Doubt using its official code, retaining its PPO training framework and reward implementation. We use Qwen3-8B-Base with the same HotpotQA training data and evaluation protocol as our main experiments. Results on HotpotQA and the average across six OOD datasets are shown below.

EvaluationMethodAcc. (%) ↑AUROC ↑Brier ↓ECE ↓
HotpotQARewarding Doubt53.00.7280.2230.129
HotpotQAIPoET63.30.8000.1840.107
OOD averageRewarding Doubt53.80.7330.2110.202
OOD averageIPoET62.30.7380.1700.146

IPoET achieves higher accuracy and AUROC, together with lower Brier scores and ECE, in both settings. Its HotpotQA AUROC improves by 0.072, while OOD Brier and ECE decrease by 0.041 and 0.056, respectively. These results support the effectiveness of our iterative policy–estimator training approach for improving confidence estimation alongside task performance.

We will incorporate this comparison and its reproduction details into the manuscript and update the reference to the published ICLR 2026 version.

[3] Bani-Harouni et al. Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models. ICLR 2026.

Reviewer_7m8cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits estimator-based and verbalized confidence estimation for LLMs. It finds that an independent confidence estimator can outperform RL-trained verbalized confidence when estimator overfitting is controlled. Building on this observation, the authors propose IPoET, which alternates policy optimization using estimator-derived confidence feedback with estimator updates on fresh policy rollouts. Experiments on question answering and mathematical reasoning with Qwen and Llama backbones show improved in-domain confidence estimation over the evaluated baselines, with more mixed but generally competitive out-of-domain results.

Critical strengths
  1. The reassessment of estimator-based confidence is useful in its own right. I appreciate that the paper examines estimator training dynamics rather than relying on a single training endpoint. The study makes a convincing case that overfitting can substantially affect the comparison in the evaluated setting, and that a carefully trained independent estimator is a strong baseline. This is a practical lesson for confidence-estimation research, even apart from the proposed method.
  2. The feedback loop is well motivated, and the empirical gains are encouraging. Using confidence-estimation errors to influence generation, rather than only scoring completed responses, is an interesting direction. Updating the estimator as the policy changes also has a clear rationale. Importantly, Table 1 reports improvements over RLVR + Estimator in both AUROC and Brier score across all four in-domain settings, so the contribution is not based solely on outperforming weak confidence baselines. The stage-wise analysis and comparison with joint training provide additional, useful support for the design.
Critical weaknesses
  1. The contribution of estimator feedback is not sufficiently isolated. My main concern is whether the gains come from estimator feedback or simply continued policy training followed by estimator refreshing. Step-aligned RLCR uses a different objective, and Figure 3 does not isolate the estimator-derived reward. A matched control removing only this reward is needed to substantiate the attribution to policy–estimator coupling.
  2. The role of reasoning remains unclear. The paper argues that IPoET makes reasoning more informative for correctness assessment, but the analyses do not isolate what reasoning traces contribute beyond the question and final answer. The results support the training recipe more directly than the claimed reasoning–representation synergy.
Questions for authors
  1. Does IPoET still outperform continued RLVR when initialization, policy-update and rollout budgets, and estimator refreshing are matched, with only RestR_{\mathrm{est}} removed?
  2. On the same rollouts, how do estimators trained with versus without reasoning traces compare? Does the benefit of including reasoning increase after the IPoET policy update?
  3. How consistent are the gains over RLVR + Estimator across individual OOD datasets? Per-dataset results would help clarify whether the reported averages reflect broad improvements or a few favorable tasks.
6 comments

We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate your recognition of the practical value of our estimator training analysis, the rationale behind IPoET, and the improvements over RLVR + Estimator. We address each concern in detail below.

Weakness 1 & Question 1. Isolating the contribution of estimator feedback.

To isolate estimator feedback, we conduct an RLVR + Estimator (step-aligned) control that removes the estimator-derived confidence reward while keeping all other training conditions matched. The control continues RLVR using the correctness reward alone, followed by the same estimator-update procedure on fresh rollouts from its own policy. This comparison uses Qwen3-8B-Base as the backbone and HotpotQA as the training dataset.

EvaluationMethodAcc. (%) ↑AUROC ↑Brier ↓ECE ↓
HotpotQARLVR + Estimator (step-aligned)60.40.7750.1970.124
HotpotQAIPoET (ours)63.30.8000.1840.107
OOD averageRLVR + Estimator (step-aligned)56.60.7510.1840.164
OOD averageIPoET (ours)62.30.7380.1700.146

As shown in the table, IPoET achieves lower Brier scores and ECE on HotpotQA and averaged over the six OOD datasets, together with higher HotpotQA AUROC. Its accuracy is also 2.9 and 5.7 percentage points higher, respectively. Continued RLVR training also reduces accuracy relative to the originally reported RLVR + Estimator baseline. Taken together, these results support the benefit of estimator-derived feedback beyond additional training and estimator refreshing alone, yielding stronger calibration while preserving task performance.

We will include this comparison and its training details in the manuscript.

Weakness 2 & Question 2. The role of reasoning traces.

Prior work provides evidence that reasoning contains useful signals for confidence estimation. Yoon et al. [1] show that removing alternative exploration and backtracking degrades calibration and discrimination despite little change in answer accuracy. Podolak and Verma [2] demonstrate that an external reader can recover uncertainty signals from reasoning traces alone, while ReProbe [3] shows that reasoning-step hidden states support trained probes for correctness assessment.

These findings motivate conditioning our estimator on the complete response, allowing it to use the reasoning behind an answer when assessing correctness. Figure 3 further shows that policy optimization improves confidence estimation while the estimator remains frozen. This indicates that changes in the generated responses can improve confidence estimation without additional estimator fitting, consistent with our objective of optimizing reasoning to better support correctness assessment.

Additional comparison on the same rollouts. To examine reasoning’s contribution beyond the question (Q) and final answer (A), we compare Q+A estimators with Q+R+A estimators that also use the reasoning trace (R). For each policy, we train the additional Q+A estimator on the same underlying HotpotQA training rollouts as its Q+R+A counterpart, using verified answer correctness as supervision but excluding reasoning from the input. We use the same evaluation datasets and protocol as our main experiments. The results are shown below.

EvaluationPolicyEstimator inputAUROC ↑Brier ↓ECE ↓
HotpotQARLVRQ+A0.7810.1880.081
HotpotQARLVRQ+R+A0.7570.1950.087
HotpotQAIPoETQ+A0.7670.1860.067
HotpotQAIPoETQ+R+A0.8000.1840.107
OOD averageRLVRQ+A0.6530.2420.260
OOD averageRLVRQ+R+A0.7240.1840.184
OOD averageIPoETQ+A0.6420.2360.241
OOD averageIPoETQ+R+A0.7370.1700.146

We define the reasoning benefit as Q+R+A − Q+A for AUROC (higher is better), and Q+A − Q+R+A for Brier and ECE (lower is better). The following table summarizes the change in this benefit from RLVR to IPoET. Positive values indicate a larger contribution from reasoning after the policy update.

EvaluationChange in AUROC benefitChange in Brier benefitChange in ECE benefit
HotpotQA+0.057+0.009−0.035
OOD average+0.025+0.009+0.020

The benefit of including reasoning increases in five of the six metric–setting comparisons, including all three OOD metrics. Thus, the observed advantage of using the complete response over Q+A generally becomes more pronounced after IPoET. Together with Figure 3, these results provide supporting evidence that post-update reasoning is more useful for correctness discrimination, particularly on unseen tasks. We will incorporate these results and the experimental details into the manuscript.

[1] Yoon et al. Reasoning Models Better Express Their Confidence. NeurIPS 2025.

[2] Podolak and Verma. Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMs. UncertaiNLP 2025.

[3] Ni et al. ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models. ACL 2026.

Question 3. Per-dataset OOD results.

We provide the per-dataset comparisons below to clarify the consistency of IPoET’s improvements over RLVR + Estimator. Each entry reports RLVR + Estimator → IPoET, with the better value in bold.

HotpotQA training, Qwen3-8B-Base

OOD datasetAUROC ↑Brier ↓ECE ↓
CommonsenseQA0.809 → 0.7970.185 → 0.1260.346 → 0.234
GPQA0.542 → 0.5790.275 → 0.2690.125 → 0.135
GSM8K0.739 → 0.7880.154 → 0.1550.257 → 0.265
MATH-5000.791 → 0.7720.184 → 0.1750.113 → 0.056
SimpleQA0.616 → 0.6400.129 → 0.1320.120 → 0.087
TriviaQA0.846 → 0.8490.178 → 0.1620.145 → 0.101

HotpotQA training, Llama-3.1-8B-Instruct

OOD datasetAUROC ↑Brier ↓ECE ↓
CommonsenseQA0.754 → 0.7530.216 → 0.2120.238 → 0.228
GPQA0.560 → 0.5760.237 → 0.2310.135 → 0.122
GSM8K0.783 → 0.7300.183 → 0.2050.251 → 0.272
MATH-5000.788 → 0.7630.196 → 0.2060.087 → 0.085
SimpleQA0.736 → 0.7480.091 → 0.0860.130 → 0.099
TriviaQA0.853 → 0.8500.156 → 0.1520.122 → 0.107

Big-Math training, Qwen3-8B-Base

OOD datasetAUROC ↑Brier ↓ECE ↓
HotpotQA0.677 → 0.7030.252 → 0.2450.173 → 0.165
CommonsenseQA0.805 → 0.7720.177 → 0.1680.313 → 0.297
GPQA0.592 → 0.6050.256 → 0.2480.100 → 0.097
SimpleQA0.580 → 0.6180.195 → 0.1710.266 → 0.251
TriviaQA0.766 → 0.7900.207 → 0.1990.134 → 0.107

Big-Math training, Llama-3.1-8B-Instruct

OOD datasetAUROC ↑Brier ↓ECE ↓
HotpotQA0.570 → 0.5700.267 → 0.2550.153 → 0.098
CommonsenseQA0.601 → 0.6120.250 → 0.2270.261 → 0.222
GPQA0.591 → 0.5680.224 → 0.2350.107 → 0.094
SimpleQA0.596 → 0.6120.213 → 0.1900.368 → 0.344
TriviaQA0.724 → 0.7490.221 → 0.2240.190 → 0.198

The per-dataset results show improvements across all three confidence-estimation metrics, with particularly consistent gains in Brier score and ECE. These gains span diverse OOD tasks and both backbones, supporting IPoET’s ability to improve confidence estimation beyond the training domain.

Reviewer_2qXoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the comparison between estimator-based and verbalization-based confidence estimation for LLMs. The authors first show that a separately trained confidence estimator can beat RL-trained verbalized confidence when the estimator is not trained for too long, and argue that estimator overfitting explains why prior work reached the opposite conclusion. Based on this finding, they propose IPoET, which alternates between two steps. The policy is updated with an extra reward computed from the frozen estimator's confidence, and the estimator is then retrained on rollouts from the updated policy. The method is tested on various question answering and mathematical reasoning benchmarks, both in-domain and out-of-domain.

Critical strengths
  • The paper addresses a timely and practically relevant problem, and the empirical study in Section 2 on estimator overfitting is an interesting observation.
  • The overall writing is clear and easy to follow, and the method is simple.
Critical weaknesses
  • One of my main concerns is the lack of comparison to prior work. In Section 5, the authors say that existing hidden-state methods only score fixed outputs, whereas IPoET turns the estimates into feedback for policy training. However, to my understanding, RLCM (Wang et al., 2026) already uses a hidden-state confidence probe as a reward during RL training, which seems quite close to the core idea of this paper. It would be helpful if the authors could explain how IPoET differs from this work and, if possible, include it as a baseline.
  • I am also not sure whether the comparison between IPoET and RLVR + Estimator is fair. If I understood correctly, IPoET gets one more round of policy RL and a second round of estimator training on top of RLVR + Estimator, while the step-aligned control is only provided for RLCR. A RLVR + Estimator baseline with the same total number of policy updates and the same estimator training would show whether the gain comes from the estimator-derived reward or simply from more training. Please correct me if I misunderstood the setup.
  • Regarding Section 2, to my understanding the paper shows that overfitting can happen when the estimator is trained for long (Figure 1b), but it does not show that the estimator baselines in prior work were actually in this regime, so the explanation of the prior discrepancy is still a hypothesis. Also, the gap between RLVR + Estimator and RLCR is quite large and goes in the opposite direction of what the RLCR paper reports, so I wonder whether the reimplemented RLCR reproduces the numbers of the original paper under the same setting.
  • The estimator has the same architecture as the policy, so the number of parameters at inference time is doubled. The paper reports inference time, but it would be helpful to also discuss the memory cost, and whether a lightweight probe on the policy's own hidden states would work similarly.
  • Some training details are missing. For example, the 1x estimator training budget is not defined in steps or epochs, the number of policy updates per round for HotpotQA is not reported, and the initialization of the estimator is not stated. Please let me know if I have overlooked these details.
Questions for authors

Why are supervised fine-tuning based verbalized confidence methods not included as baselines? To my understanding, the paper cites training-based approaches such as SaySelf and ConfTuner, but RLCR is the only trained verbalized baseline in Table 1.

5 comments

We thank the reviewer for recognizing the practical importance of our work and the insights from our study of estimator training. We also appreciate the positive comments on the paper’s clarity and the method’s simplicity. Below, we address each concern in turn and provide further clarification of our approach and experiments.

Weakness 1. Comparison with RLCM.

We agree that RLCM already uses hidden-state confidence to guide policy training, and our statement in Section 5 should be qualified accordingly. The main differences between RLCM and IPoET are:

  • Iterative training. RLCM updates its confidence probe online alongside policy training. IPoET instead freezes the entire independent estimator during each policy-update phase and subsequently refits it on fresh rollouts. This provides a fixed confidence-scoring function during policy optimization while allowing adaptation between phases. Our controlled comparison in Table 2 shows that this alternating design improves confidence estimation over joint training within our framework.
  • Confidence estimation and reward design. RLCM uses a lightweight probe of intermediate policy hidden states, supervised by Monte Carlo correctness estimates, to construct a confidence-margin reward. IPoET trains a separate full-model estimator on complete reasoning traces and answers using verified answer correctness and uses the frozen estimator’s Brier error as policy feedback. Although RLCM also evaluates a Final-Brier variant, it retains the online probe-based estimator and therefore differs from IPoET’s training design.

Beyond the method itself, we investigate how estimator training budgets affect in-domain and out-of-domain generalization in Section 2.2, and isolate the contributions of policy and estimator updates in Figure 3. Together with the joint-versus-iterative comparison, these analyses reflect a distinct emphasis on estimator optimization and policy–estimator interaction.

Additional comparison. We reproduce RLCM using its official implementation, preserving the original reward and Monte Carlo sampling design. The experiments use Qwen3-8B-Base as the backbone, with the same HotpotQA training data, common policy-training settings, and evaluation protocol as our main experiments. The results are summarized below.

EvaluationMethodAcc. (%) ↑AUROC ↑Brier ↓ECE ↓
HotpotQARLCM63.10.5850.2680.174
HotpotQAIPoET (ours)63.30.8000.1840.107
OOD averageRLCM62.90.5900.2750.251
OOD averageIPoET (ours)62.30.7380.1700.146

IPoET improves AUROC by 0.215 on HotpotQA and 0.148 across the six OOD datasets, while also achieving lower Brier scores and ECE in both settings. Consistent with our internal ablations, these results support the effectiveness of iterative policy–estimator training in improving confidence discrimination and calibration.

We will cite and discuss RLCM, incorporating these methodological distinctions, comparative results, and reproduction details into the manuscript.

Weakness 2. Comparison under matched training budgets.

The originally reported RLVR + Estimator results use the checkpoint with the best held-out in-domain validation performance. To address whether IPoET’s gains arise from additional training, we add an RLVR + Estimator (step-aligned) control with the same total number of policy updates and the same estimator training as IPoET. This comparison uses Qwen3-8B-Base as the backbone and HotpotQA as the training dataset. The results are shown below.

EvaluationMethodAcc. (%) ↑AUROC ↑Brier ↓ECE ↓
HotpotQARLVR + Estimator (step-aligned)60.40.7750.1970.124
HotpotQAIPoET (ours)63.30.8000.1840.107
OOD averageRLVR + Estimator (step-aligned)56.60.7510.1840.164
OOD averageIPoET (ours)62.30.7380.1700.146

Under matched training budgets, IPoET achieves lower Brier scores and ECE in both settings, together with higher HotpotQA AUROC. Its accuracy is also 2.9 and 5.7 percentage points higher on HotpotQA and the OOD average, respectively. Continued RLVR training yields lower accuracy than the original RLVR + Estimator baseline in this experiment. These results support the benefit of estimator-derived feedback beyond additional training alone, yielding stronger calibration while preserving task performance.

We will include this comparison and its training details in the manuscript.

Weakness 3. Estimator overfitting and RLCR reproduction.

Our empirical study focuses on the sensitivity of estimator performance to training control. The budget experiments show that larger training budgets lower training Brier scores but can degrade held-out confidence estimation. This identifies a training risk that can affect comparisons between estimator-based and verbalized approaches. In particular, evaluating only the final checkpoint may underestimate an estimator’s achievable performance. Regarding the original RLCR baseline, the official release provides estimator checkpoints and evaluation support, but we could not find the corresponding estimator-training implementation needed to determine whether it was overfit. We will revise the wording to distinguish this observed training sensitivity from our interpretation of prior performance differences.

For RLCR, we follow its original training method and evaluation protocol, using the more recently released Qwen3-8B-Base backbone, as in IPoET. We tune its hyperparameters specifically for this backbone and select the best-performing configuration on held-out validation data among those evaluated. Under this shared setting, controlling estimator training as discussed in Section 2.2 yields an RLVR + Estimator baseline that outperforms RLCR, giving the opposite ranking to that reported in the original paper.

Weakness 4. Memory cost and lightweight probes.

We will supplement the inference-time results with memory costs. Regarding lightweight probes, prior studies report that those trained on frozen features underperform estimators that also adapt the underlying representations [1, 2]. These findings motivated us to prioritize a separately fine-tuned estimator for confidence quality, rather than adopt a lightweight probe as the default architecture. This rationale is briefly discussed under “Estimator-based approaches” in Section 2.1, and we will expand that discussion to better explain our estimator choice.

Weakness 5. Training details.

We will supplement the manuscript with the following details.

Training budget. Estimator training uses eight GPUs, a per-device batch size of 1, and gradient accumulation over 8 microsteps, giving an effective batch size of 64 responses. The 1× configuration uses 312 optimizer updates, and the update budget scales proportionally with the number of rollouts per question. Policy training uses 312 updates per round on HotpotQA and 200 on Big-Math, consistent with the RLCR experimental setup.

Estimator initialization. The first estimator’s backbone is initialized from the corresponding pretrained model (Qwen3-8B-Base or Llama-3.1-8B-Instruct), with a linear regression head randomly initialized using PyTorch’s defaults. Subsequent rounds continue training from the preceding estimator checkpoint using fresh rollouts from the updated policy.

Question. Comparison with supervised verbalized confidence methods.

We chose RLCR because it optimizes reasoning about answers and confidence through reinforcement learning, providing a closer match to IPoET’s policy training than supervised fine-tuning.

Additional comparison with ConfTuner. Since ConfTuner outperforms SaySelf on average in confidence estimation [3], we select it for additional evaluation. We evaluate the authors’ released checkpoint, which was trained on HotpotQA using Llama-3.1-8B-Instruct, matching the backbone and training dataset of our corresponding experiments. We follow ConfTuner’s official prompting setup and use the same evaluation datasets and protocol as our main experiments.

EvaluationMethodAcc. (%) ↑AUROC ↑Brier ↓ECE ↓
HotpotQAConfTuner38.40.6140.2860.224
HotpotQAIPoET (ours)64.60.7810.1880.101
OOD averageConfTuner56.10.6820.2160.221
OOD averageIPoET (ours)51.50.7370.1820.152

IPoET improves AUROC by 0.167 on HotpotQA and 0.055 averaged over the six OOD datasets, while achieving lower Brier scores and ECE in both settings. These results demonstrate that IPoET achieves stronger confidence discrimination and calibration than this supervised verbalized-confidence baseline, complementing our comparison with RLCR. We will incorporate the additional results and evaluation details into the manuscript.

[1] Kapoor et al. Large Language Models Must Be Taught to Know What They Don’t Know. NeurIPS 2024.

[2] Damani et al. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. ICLR 2026.

[3] Li et al. ConfTuner: Training Large Language Models to Express Their Confidence Verbally. NeurIPS 2025.

Dear Reviewer,

Thank you for your valuable comments and suggestions. We hope that our responses have addressed your concerns and provided further clarification on the points raised in your review. We would appreciate your reconsideration of the assessment in light of our response and any further feedback or discussion.

Thank you for your time and consideration.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful reading, constructive feedback, and encouraging comments. After considering our rebuttal, all three reviewers gave our work a positive overall evaluation. Reviewers 2qXo and 5ufE raised their scores, noting that most concerns had been addressed, while Reviewer 7m8c reaffirmed their support and recognized that the new results provide stronger evidence for the proposed method. In this consolidated summary, we outline the strengths the reviewers identified and explain how we have resolved all major concerns through methodological clarifications and additional experiments.

We are pleased to know that the reviewers recognized the insights from our empirical study, the value of IPoET’s iterative training design, and its improvements over strong baselines. They also acknowledged our method’s simplicity and the comprehensive analyses supporting its design. In particular, Reviewer 7m8c highlighted IPoET’s consistent gains over the strong RLVR + Estimator baseline across all four in-domain settings.

Below, we summarize how our clarifications and additional experiments resolved the reviewers’ main concerns.

  • Methodological distinctions and additional baselines (2qXo, 5ufE). We clarified IPoET’s iterative training design and its motivation: using a frozen independent estimator to guide response generation, followed by estimator refinement on the updated policy’s outputs. We explained the distinctions from related methods and provided additional comparisons with RLCM, Rewarding Doubt, and ConfTuner. These comparisons show that IPoET achieves higher AUROC and lower Brier scores and ECE than each baseline in-domain and on average across the six OOD datasets.
  • Contribution of estimator feedback beyond additional training (2qXo, 7m8c). We conducted a step-aligned RLVR + Estimator control that removes the estimator-derived confidence reward while keeping all other training conditions matched. IPoET achieves lower Brier scores and ECE on HotpotQA and averaged over the six OOD datasets, together with higher HotpotQA AUROC. Its accuracy is also 2.9 and 5.7 percentage points higher, respectively. Continued RLVR training reduces accuracy relative to the originally reported RLVR + Estimator baseline. These results support the benefit of estimator-derived feedback beyond additional training and estimator refreshing alone, yielding stronger calibration while preserving task performance.
  • Role of reasoning and OOD generalization (7m8c). Additional comparisons between estimators trained with and without reasoning traces, using the same underlying rollouts, show a larger observed reasoning benefit after IPoET in five of six metric–setting comparisons, including all three OOD metrics. Together with Figure 3, these findings provide supporting evidence for reasoning’s role in confidence estimation. To assess the consistency of OOD improvements, we also provided per-dataset comparisons across both backbones, showing gains across diverse tasks, particularly in Brier score and ECE.
  • Interpretation and reproducibility (2qXo, 5ufE). We clarified our empirical findings on estimator-training sensitivity and how they may help explain previously reported performance differences. Our responses also provided details on the RLCR reproduction protocol, training budgets, and estimator initialization, and explained the rationale for choosing a full-model estimator over a lightweight probe.

In summary, our work contributes both a practical reassessment of estimator-based confidence estimation and an effective iterative policy–estimator training framework. The clarifications and additional validations have not only resolved the reviewers’ major concerns but also strengthened the empirical foundation of these contributions. By connecting confidence-guided response generation with estimator adaptation, IPoET offers a practical approach to improving confidence discrimination and calibration across tasks and model families.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.