- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies concentrated confidence priors as a limitation of confidence-aware RL for large reasoning models and proposes CalibSFT to address this issue. CalibSFT constructs confidence targets by combining question-level success rates with response-level correctness, uses target-balanced sampling to improve confidence exploration, and applies correctness-conditional supervision to avoid learning incorrect reasoning. The paper provides theoretical analyses of gradient suppression and calibration risk. Experiments across multiple benchmarks and multiple RL methods demonstrate improvements in calibration and discrimination while maintaining comparable accuracy.
Critical strengths
-
The paper identifies concentrated confidence priors as a potential limitation of confidence-aware RL and connects this phenomenon to insufficient on-policy exploration. This provides a useful perspective beyond designing calibration rewards. The theoretical analysis and observed training dynamics help motivate the proposed approach.
-
The experiments demonstrate that CalibSFT improves calibration and discrimination across five confidence-aware RL algorithms on both in-distribution and out-of-distribution benchmarks, while maintaining comparable accuracy. The additional evaluations across confidence formats and model families further support the practical applicability of the method.
Critical weaknesses
-
The central motivation is that concentrated confidence priors limit exploration during on-policy RL, and CalibSFT improves calibration by broadening confidence support. However, CalibSFT simultaneously introduces new confidence targets, target-balanced sampling, and additional supervised training. It is therefore unclear whether the improvements primarily come from better exploration or simply from the additional confidence supervision. In particular, Table 14 compares balanced and unbalanced sampling only at the SFT stage. Since the main motivation concerns downstream RL, comparing these two initializations after identical RL training would provide more direct evidence.
-
Proposition 1 establishes gradient suppression for low-probability actions under a categorical softmax policy. While the result is valid under its assumptions, the analysis is relatively standard and does not directly characterize optimization in autoregressive language models with shared parameters. It remains unclear whether the same mechanism is the primary bottleneck in practical confidence-aware RL. Additionally, Proposition 3 establishes proper-scoring optimality at the question level, whereas the model predicts confidence conditioned on the question, reasoning trace, and answer. Overall, the theoretical results provide useful motivation but do not fully explain why CalibSFT should improve downstream RL optimization.
Questions for authors
-
Can the authors compare balanced and unbalanced CalibSFT after applying the same downstream confidence-aware RL algorithm? Table 14 shows that balanced sampling substantially increases confidence support, but it is unclear whether this translates into better RL performance. This comparison would directly help validate the central exploration hypothesis.
-
How does CalibSFT compare against simpler exploration strategies, such as confidence-token entropy regularization or increased sampling temperature during RL? Such baselines would help clarify whether the gains require the proposed SFT initialization.
-
Can the authors provide empirical evidence of the gradient-suppression mechanism in actual language-model training, such as confidence-token gradient statistics or optimization trajectories? This would help connect Proposition 1 to the practical RL setting.
-
Given that Proposition 3 establishes question-level rather than response-level optimality, can the authors clarify how the bias discussed in Appendix B.3 affects the final response-level calibration and discrimination results?
Response to Reviewer pbAu
Thank you for the insightful comments and constructive suggestions. We address each concern below.
1. [W1, Q1] Effect of target-balanced sampling on RL
To verify whether target-balanced sampling benefits RL, we trained RLCR from the unbalanced CalibSFT model using the same RL settings as CalibSFT → RLCR. The table reports the average results over the 8 ID and 8 OOD datasets.
The results show that the unbalanced variant performs worse than the proposed method with balanced sampling, especially on OOD datasets (ECE 24.63 vs. 18.20). Although the unbalanced variant has a higher ID AUROC after SFT (81.74 vs. 79.93 in Table 18), it has a lower one after RL (85.30 vs. 86.32). This suggests the benefit of target-balanced sampling for RL, which is consistent with our motivation that broader support facilitates confidence exploration during RL. We have added these results to Appendix D.7.2 in the revised manuscript.
2. [Q2] Comparison with entropy regularization and higher temperature
Following this suggestion, we added entropy regularization on the confidence tokens to RLCR, i.e., L=LRLCR−βHconf, where Hconf is the entropy of the confidence tokens and β=10−4. Since our RL rollouts already use temperature 1.0, we compare with this regularization instead of a higher temperature. We kept the other settings of RLCR in Table 1. The table reports the average results over the 8 ID and 8 OOD datasets.
The results show that entropy regularization increases confidence diversity and modestly reduces ECE, but still underperforms CalibSFT → RLCR. This suggests that diverse confidence alone does not guarantee calibration. CalibSFT instead broadens the confidence around calibrated targets, which makes the broader support useful for RL. We have added these results to Appendix D.8.
3. [W2, Q3] Empirical evidence of gradient suppression
Following this suggestion, we measured the gradient on the confidence tokens in actual RL training. We randomly selected 512 RL training questions and sampled 8 rollouts for each question from the initial models of RL with and without CalibSFT. We then computed the policy gradient with the RLCR reward on the logits of the confidence tokens. We report the confidence token entropy Hconf, the gradient norm ∥∇ηJ∥ on the confidence logits η, and the number of distinct confidence values per question ∣Cx∣.
The results show that without CalibSFT, the model tends to put most of the probability on a few confidence values. As a result, the gradient norm on the confidence logits is much smaller than that with CalibSFT (0.095 vs. 0.460), which is consistent with Proposition 1. For the optimization trajectories, Appendix D.1 shows that the confidence token entropy stays low during RL. These results suggest that the gradient suppression also appears in actual RL training. We have added this analysis to Appendix B.1.
4. [W2, Q4] Effect of the bias in Proposition 3
Yes, our targets are calibrated at the question level, since the correctness probability of a single response cannot be observed directly. Specifically, our target ci∗=λqx+(1−λ)zi mixes two signals: the correctness zi∈0,1 of each response and the success rate qx of the question. The former is unbiased but has high variance, whereas the latter reduces variance at the cost of introducing bias toward qx (Appendix B.3). The ablation study of λ in Appendix D.7.3 shows that λ=0.5 can balance the bias–variance trade-off, outperforming λ=0 (correctness only) and λ=1 (success rate only) on AUROC and Brier score. Proposition 3 provides a question-level theoretical analysis that offers mathematical insight into the proposed method, whose effectiveness is empirically validated in the experiments. We have clarified this in Appendix B.3.