acceptodds
Under review as a conference paper at ICLR 2027

On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models

Abstract

Large reasoning models (LRMs) often suffer from overconfidence when expressing their uncertainty. Confidence-aware reinforcement learning (RL) offers a promising way to optimize calibration. However, it relies on on-policy rollouts and is thus constrained by the model's pre-RL confidence distribution, which we term *confidence prior*. In this work, we reveal that off-the-shelf LRMs exhibit a confidence prior heavily concentrated on a few high values, which persists throughout RL. Theoretically, we prove that this concentration suppresses policy gradient updates for rarely sampled confidence values and inflates the lower bound on expected Brier risk. To overcome this exploration bottleneck, we propose **CalibSFT**, a plug-and-play supervised fine-tuning stage that shapes a calibrated confidence prior with broad support before RL. For each question, CalibSFT constructs confidence targets combining its success rate with response-level correctness, which provably preserves proper-scoring optimality, and then balances training responses across the confidence spectrum to enable diverse confidence exploration during RL. To learn from incorrect responses without imitating their reasoning, CalibSFT introduces correctness-conditional supervision, guiding confidence across all responses while supervising reasoning only on correct ones. Across 16 mathematical and general reasoning benchmarks, incorporating CalibSFT reduces calibration errors and improves discrimination across five representative RL algorithms while preserving comparable accuracy. Furthermore, CalibSFT delivers practical benefits for downstream selective prediction and model routing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Verbalized Confidence, Reasoning LLMs, Reinforcement Learning, Supervised Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pbAuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies concentrated confidence priors as a limitation of confidence-aware RL for large reasoning models and proposes CalibSFT to address this issue. CalibSFT constructs confidence targets by combining question-level success rates with response-level correctness, uses target-balanced sampling to improve confidence exploration, and applies correctness-conditional supervision to avoid learning incorrect reasoning. The paper provides theoretical analyses of gradient suppression and calibration risk. Experiments across multiple benchmarks and multiple RL methods demonstrate improvements in calibration and discrimination while maintaining comparable accuracy.

Critical strengths
  1. The paper identifies concentrated confidence priors as a potential limitation of confidence-aware RL and connects this phenomenon to insufficient on-policy exploration. This provides a useful perspective beyond designing calibration rewards. The theoretical analysis and observed training dynamics help motivate the proposed approach.

  2. The experiments demonstrate that CalibSFT improves calibration and discrimination across five confidence-aware RL algorithms on both in-distribution and out-of-distribution benchmarks, while maintaining comparable accuracy. The additional evaluations across confidence formats and model families further support the practical applicability of the method.

Critical weaknesses
  1. The central motivation is that concentrated confidence priors limit exploration during on-policy RL, and CalibSFT improves calibration by broadening confidence support. However, CalibSFT simultaneously introduces new confidence targets, target-balanced sampling, and additional supervised training. It is therefore unclear whether the improvements primarily come from better exploration or simply from the additional confidence supervision. In particular, Table 14 compares balanced and unbalanced sampling only at the SFT stage. Since the main motivation concerns downstream RL, comparing these two initializations after identical RL training would provide more direct evidence.

  2. Proposition 1 establishes gradient suppression for low-probability actions under a categorical softmax policy. While the result is valid under its assumptions, the analysis is relatively standard and does not directly characterize optimization in autoregressive language models with shared parameters. It remains unclear whether the same mechanism is the primary bottleneck in practical confidence-aware RL. Additionally, Proposition 3 establishes proper-scoring optimality at the question level, whereas the model predicts confidence conditioned on the question, reasoning trace, and answer. Overall, the theoretical results provide useful motivation but do not fully explain why CalibSFT should improve downstream RL optimization.

Questions for authors
  1. Can the authors compare balanced and unbalanced CalibSFT after applying the same downstream confidence-aware RL algorithm? Table 14 shows that balanced sampling substantially increases confidence support, but it is unclear whether this translates into better RL performance. This comparison would directly help validate the central exploration hypothesis.

  2. How does CalibSFT compare against simpler exploration strategies, such as confidence-token entropy regularization or increased sampling temperature during RL? Such baselines would help clarify whether the gains require the proposed SFT initialization.

  3. Can the authors provide empirical evidence of the gradient-suppression mechanism in actual language-model training, such as confidence-token gradient statistics or optimization trajectories? This would help connect Proposition 1 to the practical RL setting.

  4. Given that Proposition 3 establishes question-level rather than response-level optimality, can the authors clarify how the bias discussed in Appendix B.3 affects the final response-level calibration and discrimination results?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pbAu

Thank you for the insightful comments and constructive suggestions. We address each concern below.

1. [W1, Q1] Effect of target-balanced sampling on RL

To verify whether target-balanced sampling benefits RL, we trained RLCR from the unbalanced CalibSFT model using the same RL settings as CalibSFT → RLCR. The table reports the average results over the 8 ID and 8 OOD datasets.

MethodIDOOD
ECE↓AUROC↑ECE↓AUROC↑
RLCR14.4682.3631.1769.38
Unbalanced CalibSFT → RLCR8.9285.3024.6371.26
CalibSFT → RLCR8.5986.3218.2071.31

The results show that the unbalanced variant performs worse than the proposed method with balanced sampling, especially on OOD datasets (ECE 24.63 vs. 18.20). Although the unbalanced variant has a higher ID AUROC after SFT (81.74 vs. 79.93 in Table 18), it has a lower one after RL (85.30 vs. 86.32). This suggests the benefit of target-balanced sampling for RL, which is consistent with our motivation that broader support facilitates confidence exploration during RL. We have added these results to Appendix D.7.2 in the revised manuscript.

2. [Q2] Comparison with entropy regularization and higher temperature

Following this suggestion, we added entropy regularization on the confidence tokens to RLCR, i.e., L=LRLCR−βHconf\mathcal{L} = \mathcal{L}_{\text{RLCR}} - \beta \mathcal{H}_{\text{conf}}, where Hconf\mathcal{H}_{\text{conf}} is the entropy of the confidence tokens and β=10−4\beta = 10^{-4}. Since our RL rollouts already use temperature 1.0, we compare with this regularization instead of a higher temperature. We kept the other settings of RLCR in Table 1. The table reports the average results over the 8 ID and 8 OOD datasets.

MethodIDOOD
ECE↓AUROC↑Support↑ECE↓AUROC↑Support↑
RLCR14.4682.3615.7531.1769.3818.62
RLCR + entropy regularization12.8577.7279.5028.5267.7587.38
CalibSFT → RLCR8.5986.3286.8818.2071.3194.50

The results show that entropy regularization increases confidence diversity and modestly reduces ECE, but still underperforms CalibSFT → RLCR. This suggests that diverse confidence alone does not guarantee calibration. CalibSFT instead broadens the confidence around calibrated targets, which makes the broader support useful for RL. We have added these results to Appendix D.8.

3. [W2, Q3] Empirical evidence of gradient suppression

Following this suggestion, we measured the gradient on the confidence tokens in actual RL training. We randomly selected 512 RL training questions and sampled 8 rollouts for each question from the initial models of RL with and without CalibSFT. We then computed the policy gradient with the RLCR reward on the logits of the confidence tokens. We report the confidence token entropy Hconf\mathcal{H}_{\text{conf}}, the gradient norm ∥∇ηJ∥\Vert\nabla_\eta J\Vert on the confidence logits η\eta, and the number of distinct confidence values per question ∣Cx∣\vert\mathcal{C}_x\vert.

w/o CalibSFTw/ CalibSFT
Hconf\mathcal{H}_{\text{conf}}0.111.06
∥∇ηJ∥\Vert\nabla_\eta J\Vert0.0950.460
∣Cx∣\vert\mathcal{C}_x\vert1.855.55

The results show that without CalibSFT, the model tends to put most of the probability on a few confidence values. As a result, the gradient norm on the confidence logits is much smaller than that with CalibSFT (0.095 vs. 0.460), which is consistent with Proposition 1. For the optimization trajectories, Appendix D.1 shows that the confidence token entropy stays low during RL. These results suggest that the gradient suppression also appears in actual RL training. We have added this analysis to Appendix B.1.

4. [W2, Q4] Effect of the bias in Proposition 3

Yes, our targets are calibrated at the question level, since the correctness probability of a single response cannot be observed directly. Specifically, our target ci∗=λqx+(1−λ)zic_i^* = \lambda q_x + (1-\lambda) z_i mixes two signals: the correctness zi∈0,1z_i \in \\{0, 1\\} of each response and the success rate qxq_x of the question. The former is unbiased but has high variance, whereas the latter reduces variance at the cost of introducing bias toward qxq_x (Appendix B.3). The ablation study of λ\lambda in Appendix D.7.3 shows that λ=0.5\lambda = 0.5 can balance the bias–variance trade-off, outperforming λ=0\lambda = 0 (correctness only) and λ=1\lambda = 1 (success rate only) on AUROC and Brier score. Proposition 3 provides a question-level theoretical analysis that offers mathematical insight into the proposed method, whose effectiveness is empirically validated in the experiments. We have clarified this in Appendix B.3.

Reviewer_GzEaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper attributes limited confidence exploration during calibration-aware RL to concentrated initial verbalized confidence distributions. CalibSFT constructs targets by mixing question success rates with response correctness, balances examples across target values, and supervises reasoning only on correct responses while supervising confidence on all responses. It then initializes existing RL methods. Experiments cover sixteen mathematical/general-reasoning benchmarks, five RL algorithms, alternative confidence formats, a second trained model family, and selective prediction/routing. Three propositions motivate the method under simplified conditional models.

Critical strengths
  1. Broad, quantitatively substantial empirical benefits. Table 1 improves average Brier, ECE, and AUROC for every RL method on both splits while preserving or improving average accuracy. For RLCR, ID ECE falls from 14.46 to 8.59 and OOD Brier from 31.14 to 23.24. Recalculation of the per-dataset entries reproduces the reported macro averages and the 61/80 ECE-improvement count. Table 8 provides additional three-seed evidence for RLCR and DCPO; this is stronger support than a single favorable benchmark.
  2. Practical and interpretable training design. Correctness-conditional supervision addresses the real tension between learning confidence on errors and imitating incorrect solutions. Appendix D.5 tests target construction, mixing weight, sampling, and supervision choices. Confidence-support diagnostics and held-out threshold selection for selective prediction/routing connect the proposed initialization to useful deployment decisions.
Critical weaknesses
  1. The distinctive mechanism is not sufficiently isolated. Confidence SFT followed by calibration RL already appears in SaySelf and the SFT+RLCR variant. The potentially valuable contribution is therefore the particular target/balancing/supervision design. Appendix D.5 partially addresses this, but its component comparisons stop after SFT. In particular, Table 14's unbalanced model has better Brier/AUROC despite narrower support; no corresponding post-RL comparison establishes that broadening support causes the eventual gains. The theory does not close this gap: Proposition 1 concerns fixed-context confidence-action logits, whereas actual training uses shared parameters, multiple tokens, and normalized/clipped advantages. Proposition 3 preserves the question-level optimum before filtering/balancing, not calibration after token cross-entropy training. Appendix B.3 appropriately acknowledges response-level shrinkage bias. These qualifications should constrain the broader mechanism claims. This concern is about contribution attribution, not the absence of comparisons with every concurrent or arXiv paper.
  2. Important comparison provenance and total-budget accounting remain unclear. Table 3's default probability RLCR baseline differs from Table 1: ID ECE/AUROC are 14.05/81.81 versus 14.46/82.36. Table 14 is labeled DeepScaleR-Eval, but its balanced results nearly reproduce the ID macro average rather than Table 6(a), and fractional unique-value counts require an unexplained averaging dimension. The appendix does not reconcile these discrepancies. Appendix D.8 does disclose approximately 147 GPU-hours for offline collection, but the advertised 6.9% measures SFT updates alone. Equal RL steps therefore do not establish equal total resources; baseline prefiltering and amortization need explicit accounting. These are fixable reporting and evidence gaps, not evidence of fabrication. Split identifiers, cross-benchmark deduplication, and complete RL configurations would also make the central comparison auditable.
Questions for authors
  1. Does broadening confidence support improve the subsequent RL stage beyond the benefit of additional supervised training? A matched-data, matched-resource comparison of balanced and unbalanced CalibSFT initialisations followed by the same RLCR procedure would help isolate this mechanism. Do the existing results support that distinction, or should the causal interpretation be narrowed?

  2. Could you explain why the default RLCR results in Table 3 differ from Table 1, and why the balanced results in Table 14 resemble the ID aggregate despite its DeepScaleR-Eval label? Are these differences due to evaluation settings, checkpoint selection, or aggregation, and do the central comparisons hold under a consistent reporting convention?

  3. When offline response generation, SFT, and RL are all included, what is the performance–compute trade-off relative to the RL-only baseline, and how much does reuse of the collected data change it? Please also explain how evaluation questions are separated from training and how potential cross-benchmark overlap is controlled.

2 comments
AuthorsOpenReview ↗

Response to Reviewer GzEa (1/2)

Thank you for the careful review and constructive suggestions. We address each concern below.

1. [W1, Q1] Effect of target-balanced sampling on RL

To verify whether target-balanced sampling benefits RL, we trained RLCR from the unbalanced CalibSFT model using the same RL settings as CalibSFT → RLCR. The table reports the average results over the 8 ID and 8 OOD datasets.

MethodIDOOD
ECE↓AUROC↑ECE↓AUROC↑
RLCR14.4682.3631.1769.38
Unbalanced CalibSFT → RLCR8.9285.3024.6371.26
CalibSFT → RLCR8.5986.3218.2071.31

The results show that the unbalanced variant performs worse than the proposed method with balanced sampling, especially on OOD datasets (ECE 24.63 vs. 18.20). Although the unbalanced variant has a higher ID AUROC after SFT (81.74 vs. 79.93 in Table 18), it has a lower one after RL (85.30 vs. 86.32). This suggests the benefit of target-balanced sampling for RL, which is consistent with our motivation that broader support facilitates confidence exploration during RL. We have added these results to Appendix D.7.2 in the revised manuscript.

2. [W1] Clarification on the contribution of CalibSFT

We clarify that our contribution lies in the insight that the concentrated confidence prior limits RL for calibration, which can be resolved by providing broad support with CalibSFT. In the literature, SaySelf [1] uses SFT to learn consistency-based confidence and GPT-4-generated rationales, while RLCR [2] uses an optional SFT stage to teach the model to analyze its uncertainty. Neither considers the confidence prior before RL, which persists throughout on-policy confidence-aware RL. CalibSFT targets this prior, so it can serve as a plug-and-play initialization for existing confidence-aware RL methods, and it consistently improves calibration across five RL algorithms, including RLCR [2]. Compared with the targets of SaySelf, our targets also lead to a higher AUROC (79.68 vs. 57.69) and a lower Brier score (17.74 vs. 39.38) after SFT (Appendix D.7.1). We have added this discussion and made our contribution clearer in Appendix A.

3. [W1] Clarification on the theoretical results

We agree that our theoretical results rely on simplified assumptions. The propositions aim to explain the calibration problem of confidence-aware RL and to motivate our method. To check whether these conclusions hold in actual training, we provide empirical evidence below.

Proposition 1: It shows that rarely sampled confidence values receive small gradients. To check this in actual RL training, we randomly selected 512 RL training questions and sampled 8 rollouts for each question from the initial models of RL with and without CalibSFT. We then computed the policy gradient with the RLCR reward on the logits of the confidence tokens. Without CalibSFT, the model tends to put most of the probability on a few confidence values, and the measured gradient norm on the confidence logits of vanilla RLCR is much smaller than that with CalibSFT (0.095 vs. 0.460), which is consistent with Proposition 1. We also show that the confidence token entropy stays low during RL in Figure 7 of Appendix D.1. These results suggest that the gradient suppression also appears in actual RL training. We have clarified this in Proposition 1 and added the gradient analysis to Appendix B.1.

Proposition 3: It shows that our targets preserve the question-level optimum before balancing, but the model may not remain calibrated after balancing and SFT. We therefore verify the calibration after SFT empirically. As shown in Table 1, CalibSFT alone lowers the ECE of the base model on both ID and OOD datasets.

4. [W2, Q2] Clarification on result inconsistency

Thanks for the careful check.

  1. The difference comes from the seeds used in the RL training: the default RLCR baseline in Table 3 was trained with a different seed from that in Table 1. In the revised version, we have updated the baselines in Table 3 to use the same seed as Table 1. The updated results keep the conclusion that CalibSFT consistently improves all the calibration metrics. We have specified the training seed in Appendix C.1 and the results across three random seeds are reported in Appendix D.3.

  2. For Table 18, the results are averaged across 8 ID datasets, and we mislabeled them as DeepScaleR-Eval in the caption. Support and Eff. Supp. are also averaged over these datasets, so they can be fractional. We have corrected the caption and the experimental setup in Appendix D.7.2.

AuthorsOpenReview ↗

Response to Reviewer GzEa (2/2)

5. [W2, Q3] Total computational cost

Following this suggestion, we report the GPU-hours of all stages for RLCR below. Most of the extra cost comes from the CalibSFT rollouts, where n=50 responses per question let the targets in Eq. (8) cover the 0.01 grid. Since the CalibSFT model is trained only once, it can be reused as the initialization for different RL methods.

StageRLCRCalibSFT → RLCR
Prefiltering rollouts32.032.0
CalibSFT rollouts–131.2
SFT–7.9
RL101.897.7
Total133.8268.8

To check whether the gain comes from the extra compute, we continued training RLCR from 160 to 240 steps. As shown in the table below, more RL steps barely change the results of RLCR, and CalibSFT → RLCR remains much better. We have updated Appendix D.11 with the total cost and these results.

MethodIDOOD
ECE↓AUROC↑ECE↓AUROC↑
RLCR (160 steps)14.4682.3631.1769.38
RLCR (240 steps)13.7581.8432.8969.33
CalibSFT → RLCR (160 steps)8.5986.3218.2071.31

6. [W2, Q3] Data splits and training details

We clarify that DeepScaleR-Eval is randomly held out from the deduplicated DeepScaleR, and DeepScaleR-Train is used for both CalibSFT and RL (Appendix C.1). Since the overlap between open-source math training data and MATH-500 has been discussed in prior work [3], we also checked the overlap between the training questions and the test questions of the 16 benchmarks. Under exact matching, only 3 of the 500 MATH-500 questions appear in the training set. Removing these questions and near-duplicate ones results in trivial influence on the performance reported in Table 1 (<0.1%), which does not affect our conclusions. We have added this discussion and the detailed RL configurations to Appendix C.1, and will release the split IDs in the final version.

References

[1] Tianyang Xu et al. SaySelf: Teaching LLMs to express confidence with self-reflective rationales. EMNLP, 2024.

[2] Mehul Damani et al. Beyond binary rewards: Training LMs to reason about their uncertainty. ICLR, 2026.

[3] Liang Wen et al. Light-R1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. ACL, 2025.

Reviewer_JitdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the phenomenon where Large Reasoning Models (RMs) frequently exhibit overconfidence when expressing uncertainty. To address this, the authors propose CalibSFT, a plug-and-play Supervised Fine-Tuning (SFT) stage designed to calibrate the model's confidence distribution prior to Reinforcement Learning (RL). CalibSFT address this problem through mixed confidence targets, target-balanced sampling, and correctness-conditional supervision. Experimentally, the authors evaluate their method on 16 mathematical and general reasoning benchmarks. Overall, the work is quite comprehensive.

Critical strengths
  1. From the motivation and proposed method to the experiments and ablation studies, this work is fairly complete. The paper uses the confidence distributions of Qwen3-8B, Gemma4-E2B, and OLMo3-7B to introduce the problem of overconfidence in LRMs. It then proposes its own method, CalibSFT, to address this problem through mixed confidence targets, target-balanced sampling, and correctness-conditional supervision.
  2. The research problem addressed in the paper is significant. By focusing on the prior distribution of confidence levels before the reinforcement learning process begins, the paper analyzes issues such as model overconfidence and insufficient exploration of confidence. This problem has strong practical application value.
  3. The proposed method is well-designed. It does not require modifications to existing reinforcement learning algorithms or their reward functions, offering strong potential for integration. Furthermore, the proper-scoring optimality of the mixed target proposed in this work should be one of the major contributions of the paper. When designing the target confidence, it considers both the overall difficulty of the question and whether the answer is correct. In addition, it provides a complete mathematical proof.
  4. The experimental analysis is thorough. The authors conducted comprehensive experiments across 16 mathematical and general reasoning benchmarks.
Critical weaknesses
  1. In Proposition 1, confidence generation is formulated as a classification task. However, confidence generation is commonly treated as a regression task in many settings. How did the authors think about and make this choice? It is suggested to add an explanation for this part.
  2. This work discusses how LRM models are often overconfident when expressing uncertainty. However, modifying the prompt is likely to affect overall confidence scores; currently, CalibSFT appears to rely on a fixed prompt for its experimental procedures. Can CalibSFT enable the model to maintain better performance across different prompts? It is suggested that the authors provide experimental results using a wider variety of prompts.
  3. The experiments focus primarily on models such as Qwen3-8B, Gemma4-E2B, and OLMo3-7B. Do the issues identified in the paper persist across models of different sizes? Furthermore, can increasing model size help mitigate these issues? The current experimental analysis in the paper does not provide an answer to this. It is recommended to include experiments on models of varying sizes to verify the consistency of the proposed approach.
  4. The paper points out that the confidence of the base model is concentrated at a few high values, which in turn affects subsequent reinforcement learning. Therefore, CalibSFT is performed before reinforcement learning to broaden the initial confidence distribution. Could CalibSFT also be added after reinforcement learning for comparison?
Questions for authors
  1. Is this approach effective only for specific prompts, or can it be generalized to a broader range of prompts? Could the authors evaluate its performance under different prompt formulations?
  2. Can the experiments be extended to LLMs of different sizes to verify the consistency of the approach? Could larger models help mitigate this issue?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Jitd

Thank you for the positive comments and constructive suggestions. We address each concern below.

1. [W1] Confidence formulation in Proposition 1

We clarify that our choice follows from how an LLM produces verbalized confidence: it generates the confidence as tokens from its vocabulary (e.g., "0.80"), which amounts to a classification over a finite set of values rather than a regression to a continuous score. Since the model receives rewards on the values it samples during RL, Proposition 1 models the confidence as a softmax policy over these values. We have modified Section 3 in the revised manuscript to make this clearer.

2. [W2, Q1] Performance under different prompts

Following this suggestion, we evaluate CalibSFT → RLCR under prompts that differ from the training prompt, using one simpler and one more complex one. We report results on MinervaMath, AIME 2026, HotpotQA, and TriviaQA in the table below, using the same protocol as in the main experiments.

MethodPromptMinervaMathAIME 2026HotpotQATriviaQA
Pass@1↑ECE↓Pass@1↑ECE↓Pass@1↑ECE↓Pass@1↑ECE↓
RLCRDefault51.7523.0547.8123.1965.8019.7755.2423.30
CalibSFT → RLCRSimple48.6218.3458.3314.7364.408.1856.1110.38
CalibSFT → RLCRDefault51.9312.0052.6012.0466.2710.8555.2510.71
CalibSFT → RLCRComplex52.1111.6055.3114.3266.157.4055.998.43

The results show that under both additional prompts, CalibSFT → RLCR still achieves a lower ECE than RLCR on all four datasets while maintaining comparable Pass@1. This indicates that CalibSFT → RLCR generalizes to prompts not used in training. We have added the used prompts and the full results to Appendix D.4.

3. [W3, Q2] Effect of model size

3.1 Confidence prior of different model sizes

Following this suggestion, we evaluated the confidence prior of models of different scales. We evaluated Qwen3-1.7B, 4B, 8B, and 14B on DeepScaleR-Eval, with 4 responses per question. We also include Qwen3.7-Flash, which is a newer API model.

ModelMean conf (%)Accuracy (%)ECE↓AUROC↑
Qwen3-1.7B93.453.641.461.8
Qwen3-4B96.864.833.065.7
Qwen3-8B95.565.530.364.8
Qwen3-14B94.568.426.764.2
Qwen3.7-Flash (API)97.372.425.760.9

The results show that as the model size increases from Qwen3-1.7B to 14B, the accuracy rises from 53.6% to 68.4%, but the mean confidence stays above 93%. The ECE decreases but remains high, and the AUROC shows no consistent improvement. The results also hold for the newer Qwen3.7-Flash. This shows that a larger or newer model alone does not eliminate the miscalibration, and a model can hardly verbalize calibrated confidence without confidence-aware training, which is consistent with our findings in Section 3.

3.2 CalibSFT at a different model scale

Due to limited computational resources, we could not train a model larger than 8B during the rebuttal period. To verify whether CalibSFT works at a different model scale, we trained Qwen3-1.7B with RLCR and with CalibSFT → RLCR. We report the average results over the 8 ID and 8 OOD datasets in the table below.

ModelMethodIDOOD
ECE↓AUROC↑ECE↓AUROC↑
Qwen3-1.7BRLCR19.7178.3137.6965.05
Qwen3-1.7B+ CalibSFT15.6582.9819.3967.11
Qwen3-8BRLCR14.4682.3631.1769.38
Qwen3-8B+ CalibSFT8.5986.3218.2071.31

The results show that when trained with RLCR alone, the 8B model has better ECE and AUROC than the 1.7B model, so a larger model may show better calibration after RL training. CalibSFT achieves better ECE and AUROC for both models, which shows that CalibSFT works consistently across model scales. We have added these results to Appendix D.5.

4. [W4] The order of CalibSFT and RL

Following this suggestion, we add additional experiments by applying CalibSFT after RLCR (RLCR → CalibSFT). The table reports the average results over the 8 ID and 8 OOD datasets.

MethodIDOOD
ECE↓AUROC↑ECE↓AUROC↑
RLCR14.4682.3631.1769.38
CalibSFT → RLCR8.5986.3218.2071.31
RLCR → CalibSFT15.2772.7724.4057.98

The results show that RLCR → CalibSFT performs worse than CalibSFT → RLCR on both ID and OOD datasets, which is intuitive as the design of CalibSFT is to provide a calibrated confidence prior with broad support for RL to explore (see Section 4). We have added the detailed discussion and illustration to Appendix D.7.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.