acceptodds
Under review as a conference paper at ICLR 2027

Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

Abstract

Reinforcement learning with verifiable rewards (RLVR) trains reasoning models to produce correct answers, but does not ensure that their stated confidence is calibrated. The resulting models are systematically overconfident. Recent methods train calibration inside the RLVR loop by having the model state a numerical confidence alongside its answer, but they all obtain the confidence by sampling it as text. This choice imposes two costs: a sampled confidence introduces variance and in practice collapses to a handful of distinct values, and sampling makes the confidence non-differentiable, forcing the calibration loss through a scalar reward. We propose (Confidence REaDOut) to replace sampling with a deterministic readout. While RLVR optimizes correctness, CREDO reads the confidence from a dedicated token pair in the model's output distribution and trains it by differentiable regression. CREDO further turns the trained confidence into a signal for accuracy, weighting rollouts by how far confidence and outcome disagree, so that accuracy and calibration improve together. Across mathematical and code reasoning, CREDO attains the best accuracy and calibration, and the gains extend to abstention and selective prediction. Code for CREDO and all baselines is available at https://anonymous.4open.science/r/CREDO-5678.

Primary area:
foundation or frontier models, including LLMs
Keywords:
confidence calibration, large language models, reinforcement learning with verifiable rewards, uncertainty quantification, reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_ibnHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CREDO, a method for training calibrated reasoning models. Instead of sampling verbalized confidence scores, CREDO uses a differentiable readout based on the relative probabilities of two dedicated tokens. Experiments on mathematical and code reasoning demonstrate improvements in both accuracy and calibration over standard GRPO and calibration baselines such as RLCR, and DCPO.

Critical strengths
  • Simple and effective approach: Verbalized confidence suffers from issues such as entropy collapse and variance. The differentiable readout is a natural alternative to verbalized confidence which fixes some of these issues.

  • Strong empirical results: CREDO achieves substantial improvements in calibration, while interestingly also improving accuracy over standard RLVR.

  • Thorough experimental evaluation: The paper includes strong baselines, multiple ablations, different model scales, and downstream evaluations of selective prediction. I particularly appreciate the experiments disentangling the confidence readout from the training objective.

Critical weaknesses
  • Understanding the accuracy improvements. One of the most interesting findings is that CREDO substantially improves accuracy over vanilla GRPO, whereas other calibration methods (RLCR and DCPO) do not show similarly consistent gains. While the ablations identify contributing components, they do not fully explain why these improvements occur. If there is a synergy between calibration and correctness optimization, why don't other methods benefit similarly? Investigating the underlying cause of this would be useful- it could be because of potential differences in reasoning length, reward design (particularly the format reward), and training dynamics.

  • Novelty relative to Chuang et al. (2025). The confidence-token readout is adopted from [1]. The primary contributions therefore appear to be incorporating this mechanism into RLVR and introducing discrepancy weighting. A clearer discussion of the differences and what specifically enables the improvements in this setting, would strengthen the paper.

  • Limited OOD evaluation. The OOD experiments are restricted to transfer between mathematics and code. Given that evaluating additional datasets requires no further training, it would be nice to see a broad range of both reasoning and non-reasoning tasks in this evaluation.

[1]: Chuang, Y. N., Sarma, P. K., Gopalan, P., Boccio, J., Bolouki, S., Hu, X., & Zhou, H. (2024). Learning to route llms with confidence tokens. arXiv preprint arXiv:2410.13284.

Questions for authors
  • Accuracy gains: What do the authors believe drives the substantial accuracy improvements over vanilla GRPO? Why don't other calibration methods such as RLCR and DCPO benefit similarly? Could differences in reasoning length, format rewards, or training dynamics explain these gains?

  • Confidence reasoning: Does the differentiable confidence readout actually rely on the verbalized uncertainty analysis? Interestingly, removing this analysis substantially hurts accuracy while having relatively little effect on calibration, particularly for code. Since the analysis is generated after the answer, the accuracy gains must arise from training dynamics rather than a direct effect on the generated answer. It would be interesting to conduct an inference-time intervention where the confidence analysis is removed or replaced with an unrelated analysis, while keeping the model and answer fixed, to understand whether the content of the analysis meaningfully contributes to confidence estimation.

  • Comparison with Chuang et al.: see weaknesses

  • OOD Eval: see weaknesses

  • Quality of confidence reasoning: Are the generated confidence analysis better than baselines? Some qualitative evaluations of this would be helpful - for example comparing the confidence analysis of RLVR (should be bad), RLCR, etc to Credo.

  • Reasoning length: How do the average reasoning lengths compare across CREDO, GRPO, RLCR, and DCPO?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ibnH (1/5)

Dear Reviewer ibnH,

Thank you for your careful reading and for highlighting the experiments that separate the confidence readout from its training objective. Your observation that the analysis comes after the answer helped us distinguish what the model learns for future problems from what it uses to judge an answer it has already given. Below, we address the accuracy gains, the comparison with Self-REF, and transfer beyond math and code. We then examine the uncertainty analyses through fixed-answer interventions and qualitative examples.

1. Why does CREDO improve answer accuracy over GRPO?

Weakness 1

Understanding the accuracy improvements. One of the most interesting findings is that CREDO substantially improves accuracy over vanilla GRPO, whereas other calibration methods (RLCR and DCPO) do not show similarly consistent gains. While the ablations identify contributing components, they do not fully explain why these improvements occur. If there is a synergy between calibration and correctness optimization, why don't other methods benefit similarly? Investigating the underlying cause of this would be useful- it could be because of potential differences in reasoning length, reward design (particularly the format reward), and training dynamics.

Question 1

Accuracy gains: What do the authors believe drives the substantial accuracy improvements over vanilla GRPO? Why don't other calibration methods such as RLCR and DCPO benefit similarly? Could differences in reasoning length, format rewards, or training dynamics explain these gains?

1.1 How learning and using confidence contribute to accuracy

Thank you for asking why CREDO improves accuracy more consistently than RLCR and DCPO. In CREDO, the model that solves a problem also learns to assess its solution. Its confidence contributes to learning in two ways (Fig. 3).

  • Learning to assess its own reasoning. The two-token readout estimates how likely a solution is to be correct. Its training target combines the verifier’s verdict on the individual answer with the model’s success rate on the same question (Eqs. 6 and 7). Because regression updates the parameters that generate the solution, the solver gets direct feedback whenever it misjudges its own work. Learning to recognize which approaches are prone to error can also help it avoid similar mistakes on later problems.
  • Learning more from solutions the model misjudged. CREDO uses the gap between confidence and verified correctness, ∣ai−ci∣|a_i-c_i|, to weight each answer’s advantage within its group (Eq. 9). A confident mistake receives a stronger correction, and a correct solution the model doubted receives more reinforcement. This directs more of the policy update toward mistakes the model overlooked and successful approaches it has yet to recognize as reliable.

We extended the two individual ablations in Table 2 to three training seeds. We also added a fourth configuration that switches off both regression and discrepancy weighting, so we can measure the effect of regression under uniform answer weights as well.

ConfigurationMath Acc ↑Math ECE ↓Code Acc ↑Code ECE ↓
Full CREDO.752 ± .013.103 ± .013.600 ± .018.064 ± .020
Without calibration regression.713 ± .014.205 ± .047.591 ± .022.185 ± .012
Without discrepancy weighting.703 ± .025.122 ± .016.578 ± .006.058 ± .007
Without either component.692 ± .013.183 ± .026.562 ± .005.180 ± .009

Regression improves calibration and mean accuracy in both domains even under uniform answer weighting. Using the learned confidence to weight answers adds a further gain, giving full CREDO the highest mean accuracy among the four configurations.

Why are the gains less consistent for RLCR and DCPO? RLCR [2] rewards the confidence report together with the answer, while DCPO [3] rewards it separately (§2.2). Both sample a number, so variation across reports adds to the expected squared calibration error even for a fixed solution and analysis. Because the sampled numeral is discrete, calibration feedback reaches the model through a policy-gradient update based on its reward. CREDO trains its continuous score directly by regression, giving the shared solver feedback on the size and direction of its confidence error (§2.3, Eq. 5).

Taking the digit expectation at evaluation improves baseline AUROC, but CREDO still achieves better calibration in both domains (Table 1). Training still relies on rewards for sampled reports, leaving the continuous score without direct supervision. We think this indirect feedback makes it harder for the solver to anticipate its own reasoning errors, so confidence training brings less consistent accuracy gains.

AuthorsOpenReview ↗

Response to Reviewer ibnH (2/5)

A second difference is that neither original baseline weights an answer’s advantage by the gap between its confidence and verified correctness. We therefore retrained RLCR and DCPO with discrepancy weighting, using either the reported number or the digit expectation (Appendix C.3, Table 9).

Source of cic_i in the discrepancy si=∣ai−ci∣s_i=\lvert a_i-c_i\rvertRLCR math Acc ↑RLCR code Acc ↑DCPO math Acc ↑DCPO code Acc ↑
Original objective (no discrepancy weighting).682.539.715.579
Reported number.701.545.716.581
Digit expectation.697.542.690.560

Discrepancy weighting improves RLCR’s accuracy, especially on math. For DCPO it makes little difference with the reported number and lowers accuracy in both domains with the digit expectation. These uneven gains point to the quality of the confidence signal. When a score poorly estimates correctness, its distance from the verified outcome is a less reliable guide to which answers deserve a stronger update. Direct regression gives CREDO a better basis for identifying confident errors and underestimated successes. Using that distinction to weight the answer updates improves accuracy in both domains.

1.2 Could answer length, format rewards, or training dynamics explain the gain?

Question 6

Reasoning length: How do the average reasoning lengths compare across CREDO, GRPO, RLCR, and DCPO?

Answer length.

MethodMath answer tokensCode answer tokens
GRPO3,403.3 ± 314.32,874.9 ± 264.2
RLCR2,762.6 ± 187.21,648.9 ± 157.1
DCPO3,557.3 ± 344.42,853.3 ± 405.7
CREDO3,514.6 ± 140.52,354.0 ± 267.1

CREDO’s math answers are slightly longer on average than GRPO’s and shorter than DCPO’s. On code, CREDO achieves higher mean accuracy with shorter answers than either GRPO or DCPO. The accuracy gains therefore do not require consistently longer answers.

Format reward. We compared three ways of rewarding the answer in CREDO (Appendix C.4, Table 10). Here aa indicates a correct answer, FF indicates valid formatting, and β=0.5\beta=0.5 is the format bonus.

Answer rewardransr^{\mathrm{ans}}Math Acc ↑Code Acc ↑
Correctness onlyaa.723.583
Format-gated rewardF(β+a)F(\beta+a).714.589
Separate format bonus (CREDO)βF+a\beta F+a.753.614

When CREDO and GRPO both receive only a correctness reward for the answer, CREDO still achieves higher accuracy in both domains. Its advantage over GRPO therefore persists without a format bonus. We use the separate bonus because it gives the highest accuracy among the three reward designs. It preserves the full correctness reward when formatting fails, while still encouraging the requested format.

Training dynamics. We tracked how accurately the model predicted its calibration target throughout training. The table gives the mean squared error against that target in early, middle, and late training, averaged over three seeds.

Training domainConfigurationEarlyMiddleLate
MathFull CREDO.296.084.073
MathWithout calibration regression.301.118.115
MathWithout discrepancy weighting.294.090.080
CodeFull CREDO.350.101.096
CodeWithout calibration regression.394.186.176
CodeWithout discrepancy weighting.346.110.102

By mid-training, both configurations with regression predict their targets more accurately than the variant trained without it, and the difference persists through the final stage in both domains. Full CREDO and the variant with uniform weights finish with similar prediction errors on their own solutions. What distinguishes them is how they use those estimates. The policy that gives extra weight to answers it misjudged achieves higher held-out accuracy in both domains (Response 1.1).

AuthorsOpenReview ↗

Response to Reviewer ibnH (3/5)

2. What does CREDO add to the confidence tokens of Chuang et al.?

Weakness 2

Novelty relative to Chuang et al. (2025). The confidence-token readout is adopted from [1]. The primary contributions therefore appear to be incorporating this mechanism into RLVR and introducing discrepancy weighting. A clearer discussion of the differences and what specifically enables the improvements in this setting, would strengthen the paper.

Question 3

Comparison with Chuang et al.: see weaknesses

Thank you for highlighting this connection. Chuang et al. [1] introduced the two-token confidence readout we use. Self-REF learns it through supervised fine-tuning to decide when to accept an answer and when to route the question elsewhere. That objective limits what the readout can teach the solver. Its supervision comes from a fixed set of answers produced by the original model, and on a wrong answer the loss masks the answer tokens and trains the uncertainty token. The error teaches the system when to defer, but the confidence score is not used to strengthen a corrective update to the incorrect answer.

In RLVR the policy keeps changing, along with the errors it needs to recognize. CREDO therefore trains the readout on each new batch of verified solutions and uses the gap between confidence and correctness to weight each answer’s policy update. A confidently wrong answer can now teach the solver both to revise its judgment and to avoid repeating the mistake.

We compared Self-REF and CREDO on Qwen3-8B across three training seeds.

Training domainMethodAcc ↑ECE ↓Brier ↓AUROC ↑
MathSelf-REF.457 ± .006.298 ± .095.318 ± .084.528 ± .049
MathCREDO.752 ± .013.103 ± .013.086 ± .004.933 ± .007
CodeSelf-REF.450 ± .001.385 ± .013.367 ± .010.793 ± .017
CodeCREDO.600 ± .018.064 ± .020.127 ± .016.850 ± .027

With the same two-token readout, CREDO achieves higher mean accuracy and AUROC and lower mean ECE and Brier scores than Self-REF in both domains. These gains support CREDO’s approach of training the readout on the policy’s own verified rollouts and using confidence errors to guide answer updates.

3. Does calibration transfer beyond mathematics and code?

Weakness 3

Limited OOD evaluation. The OOD experiments are restricted to transfer between mathematics and code. Given that evaluating additional datasets requires no further training, it would be nice to see a broad range of both reasoning and non-reasoning tasks in this evaluation.

Question 4

OOD Eval: see weaknesses

Thank you for suggesting a broader test of transfer. We report results from the existing math-trained and code-trained checkpoints on eight datasets covering factual QA, reading comprehension, commonsense and logical reasoning, general knowledge, and biomedical QA. Each cell reports single-checkpoint accuracy ↑ / ECE ↓ / Brier ↓ / AUROC ↑. The macro average weights the eight datasets equally. Bold marks the best displayed value for each metric within each row, including ties.

Math-trained models

DatasetBaseGRPORLCRDCPOCREDO
NQ-Open.186/.757/.728/.654.210/.723/.688/.642.203/.716/.683/.654.210/.719/.707/.618.212/.311/.259/.740
BoolQ.466/.492/.480/.619.625/.323/.326/.658.616/.328/.326/.696.621/.360/.351/.653.799/.190/.149/.830
StrategyQA.395/.520/.487/.638.500/.405/.384/.708.546/.338/.321/.728.522/.405/.370/.696.535/.160/.192/.821
LogiQA 2.0.638/.303/.318/.562.731/.195/.224/.587.738/.176/.221/.553.732/.240/.244/.660.732/.144/.193/.699
MMLU-Pro.498/.448/.444/.617.615/.277/.287/.700.622/.279/.289/.713.641/.304/.303/.747.641/.113/.178/.818
MedQA.574/.373/.381/.553.689/.250/.272/.567.696/.225/.255/.602.701/.268/.272/.651.720/.167/.194/.758
MedMCQA.474/.477/.473/.549.562/.378/.386/.557.595/.337/.347/.591.614/.357/.357/.613.620/.078/.204/.740
PubMedQA.552/.356/.349/.696.542/.358/.358/.664.561/.320/.326/.678.584/.366/.355/.690.627/.163/.217/.738
Macro average.473/.466/.458/.611.559/.364/.366/.635.572/.340/.346/.652.578/.377/.370/.666.611/.166/.198/.768
Reviewer_kKWROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes CREDO, a method for calibrating reasoning models trained with reinforcement learning from verifiable rewards (RLVR). Unlike existing approaches that sample numerical confidence reports as text and optimize them through scalar rewards, CREDO derives confidence from the normalized probabilities of two reserved tokens and directly optimizes it using differentiable regression. It further uses confidence–outcome discrepancies to reweight policy-gradient updates, aiming to improve accuracy and calibration jointly. Experiments on mathematical and code reasoning tasks show improvements over GRPO, RLCR, and DCPO in accuracy, calibration, and selective prediction.

Critical strengths
  • The paper identifies two limitations of verbalized confidence: sampling variance and discretization caused by a limited set of numerical tokens. The proposed two-token probability readout is continuous, deterministic, differentiable, and naturally compatible with RLVR.
  • The evaluation covers mathematical and code reasoning, in-domain and cross-domain transfer, multiple model scales, and several strong baselines. The ablations separately examine differentiable regression, uncertainty analysis, confidence-segment credit, and discrepancy weighting.
  • CREDO improves not only standard calibration metrics but also risk–coverage and selective prediction performance.
Critical weaknesses

Limited evidence for the incremental novelty of CREDO. The main components of CREDO are closely related to existing methods. Differentiable logits-based confidence optimization has been explored in previous logits-based methods (e.g., C²GSPG, LaSeR), dedicated confidence-token readouts were introduced in Self-REF (Chuang et al., ICML 2025), and decoupled optimization with hybrid instance- and group-level calibration targets was established in DCPO (Ma et al., ICML 2026). Although the ablation studies demonstrate that these components are effective within CREDO, they do not adequately isolate the contributions of the genuinely new designs from those inherited from prior work. In particular, the strong effects of differentiable regression and segmented optimization do not establish their novelty. Meanwhile, the improvements from discrepancy weighting are evaluated mainly against verbalized-confidence baselines, rather than alternative confidence-aware optimization methods using comparable differentiable readouts. The authors should include stronger compositional baselines that integrate existing confidence-token representations, differentiable calibration losses, and confidence-guided policy optimization under matched training conditions. Such comparisons are essential to establish whether CREDO introduces a substantive methodological advance beyond an effective combination of existing techniques.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer kKWR (1/3)

Dear Reviewer kKWR,

Thank you for your careful review and for highlighting our analysis of sampled confidence and our selective prediction results. We agree that the clearest test of CREDO’s contribution is a comparison with a policy that also learns and uses confidence, and your suggestion led us to the four questions below.

  • Which ideas come from earlier work, and what is new about the way CREDO learns and uses confidence in RLVR?
  • How does CREDO compare with four additional confidence-aware baselines?
  • With the same readout and calibration objective, how does discrepancy weighting compare with advantage mixing?
  • What information in the confidence score makes it useful for weighting answers?

1. What CREDO adopts and how it uses confidence errors to train the reasoning policy

Thank you for asking us to be more explicit about what is new. A reasoning policy can become more accurate under RLVR while remaining systematically overconfident. We wanted the model’s confidence to track its improving accuracy and to help it learn from the answers whose outcomes it misjudged.

What the score measures. C²GSPG [1] calibrates the likelihood of generating a solution and uses it in policy optimization, while CAPO [2] trains the policy to assign higher likelihood to correct solutions than to incorrect ones. Since likelihood also depends on how a solution is worded, we adopt the two-token readout of Self-REF [3] to estimate the probability that the completed answer is correct. Self-REF fits this readout for routing on answers collected before supervised fine-tuning. On a wrong answer, its loss supervises only the uncertainty token, teaching the model when to defer while leaving the answer tokens themselves without a direct correction. In RLVR, the errors the policy makes change as it learns. We therefore train the readout on each new batch of verified solutions and use its predictions to give confidently wrong answers more weight in the policy update.

How the estimate is trained. DCPO [4] assigns credit separately to the answer and the confidence report. Its hybrid target tit_i combines the verifier’s verdict on an individual answer with the success rate on the same question. Averaging across answers reduces noise in the target, but the report is still a sampled number. Even with the preceding text fixed, the model can report different numbers, and its calibration feedback comes through the reward for whichever number it drew. Across different answers, the same few numbers recur, so correct and incorrect solutions often receive identical reports (Fig. 1c). With the same hybrid target and separate credit assignment, CREDO reads confidence from the relative probabilities of the high- and low-confidence tokens, ci=pH/(pH+pL)c_i=p_H/(p_H+p_L). The regression loss can then move this probability directly toward tit_i on each rollout while policy gradients train the answer and the uncertainty analysis.

How confidence guides the answer update. In standard GRPO, equally rewarded answers to the same question have the same advantage, regardless of how well the model anticipated their outcomes. CREDO uses ∣ai−ci∣\lvert a_i-c_i\rvert, the gap between confidence cic_i and verified correctness aia_i, to give greater weight within the group to answers whose outcomes the model misjudged. The weight is positive, so it preserves the direction of the original advantage while strengthening the correction for confident mistakes and the reinforcement for correct solutions the model underestimated (Eq. 9).

Our contribution to RLVR lies in explaining what limits sampled confidence reports and showing how the reasoning policy can learn from its own confidence errors.

  • Theory. We show that sampling a numerical confidence report incurs extra expected squared error whenever the number can vary for the same prefix, even if its mean is on target. Under reward-only training, the calibration loss reaches the policy indirectly because its gradient cannot pass through the discrete sampling step. These results motivate a confidence readout that is both deterministic and differentiable (§2).
  • Method. We train this readout by regression on the reasoning policy’s own verified solutions and introduce a discrepancy weight that scales each answer’s advantage according to the gap between confidence and correctness (§3). Together, these changes improve both answer accuracy and calibration over accuracy-only GRPO on math and code (Table 1).
AuthorsOpenReview ↗

Response to Reviewer kKWR (2/3)

2. Accuracy and calibration against four additional baselines

Thank you for pointing us to methods that already learn confidence from model outputs. We extended the comparison in Table 1 to include Self-REF, LaSeR [5], C²GSPG, and CAPO. The table below reports means and sample standard deviations over three seeds on Qwen3-8B.

MethodMath Acc ↑Math ECE ↓Math AUROC ↑Math Brier ↓Code Acc ↑Code ECE ↓Code AUROC ↑Code Brier ↓
Base.439 ± .001.472 ± .002.644 ± .004.453 ± .002.455 ± .001.418 ± .001.690 ± .004.395 ± .002
GRPO.706 ± .009.199 ± .027.775 ± .062.204 ± .029.581 ± .007.182 ± .070.753 ± .061.205 ± .066
RLCR.682 ± .010.159 ± .012.831 ± .009.171 ± .009.553 ± .013.075 ± .005.815 ± .014.146 ± .006
DCPO.722 ± .008.217 ± .008.848 ± .012.198 ± .003.576 ± .003.153 ± .024.776 ± .025.153 ± .024
Self-REF.457 ± .006.298 ± .095.528 ± .049.318 ± .084.450 ± .001.385 ± .013.793 ± .017.367 ± .010
LaSeR.647 ± .003.275 ± .041.824 ± .012.203 ± .029.523 ± .009.093 ± .010.829 ± .010.139 ± .004
C²GSPG.690 ± .009.158 ± .007.743 ± .026.196 ± .004.533 ± .011.235 ± .025.839 ± .037.207 ± .003
CAPO.659 ± .007.269 ± .030.777 ± .012.275 ± .024.488 ± .001.472 ± .003.747 ± .013.456 ± .004
CREDO.752 ± .013.103 ± .013.933 ± .007.086 ± .004.600 ± .018.064 ± .020.850 ± .027.127 ± .016

CREDO achieves the highest mean accuracy and AUROC and the lowest mean ECE and Brier score in both domains among the methods in this comparison. These results demonstrate that CREDO improves answer accuracy together with confidence estimation even when compared with baselines that already learn or use confidence.

3. Discrepancy weighting versus advantage mixing under matched conditions

We especially appreciate your suggestion to test a baseline that combines confidence tokens, differentiable calibration, and confidence-guided policy optimization. Table 2 examined individual components, and Appendix C.3 applied discrepancy weighting to RLCR and DCPO. For the comparison you propose, all three policies use the same two-token readout, regression loss and hybrid target, uncertainty-analysis objective, and training budget. They differ in how confidence enters the answer update.

  • Uniform weighting uses the original answer advantage AiA_i.
  • LaSeR-style advantage mixing treats a differentiable self-score as a second reward. We apply LaSeR’s mixing rule [5] to CREDO’s two-token readout, giving (1−τ)Ai+τz(ci)(1-\tau)A_i+\tau z(c_i), where z(ci)z(c_i) is the confidence score standardized across answers to the same question.
  • CREDO discrepancy weighting uses wiAiw_iA_i, giving an answer more weight when its confidence is further from its verified outcome.
Answer update, three seedsMath Acc ↑Code Acc ↑
Uniform weighting.703 ± .025.578 ± .006
LaSeR-style advantage mixing.709 ± .008.580 ± .010
CREDO discrepancy weighting.752 ± .013.600 ± .018

Discrepancy weighting gives higher mean answer accuracy than both uniform weighting and matched advantage mixing in both domains. With the readout and its calibration objective held fixed, this comparison shows the benefit of using the gap between confidence and verified correctness to determine how strongly each answer updates the policy.

AuthorsOpenReview ↗

Response to Reviewer kKWR (3/3)

4. Which information makes confidence useful for weighting answers?

We appreciate your emphasis on comparing weighting rules that have access to the same differentiable readout. Keeping the readout and its calibration objective fixed, we compared CREDO’s discrepancy weight with six alternatives on math and code, with uniform weighting as a reference.

  • Uncertainty. Use 4ci(1−ci)4c_i(1-c_i), which is highest when the model assigns equal probability to success and failure.
  • Question difficulty. Use each question’s error rate, centered across questions in the rollout batch, to assign the same weight to all its answers.
  • Errors only. Score an incorrect answer as 1 and a correct answer as 0.
  • Shuffled confidence. Shuffle confidence scores among answers to the same question before computing each answer’s discrepancy.
  • Mean weight by outcome. Within each question, average the CREDO weights separately for correct and incorrect answers. Give each answer the mean for its group.
  • Confidence rank. Replace each confidence score with its percentile rank among answers to the same question before computing the discrepancy.
Answer weighting, seed 43Math Acc ↑Code Acc ↑
CREDO discrepancy.753.614
Uniform.721.572
Uncertainty.730.599
Question difficulty.739.601
Errors only.706.583
Shuffled confidence.674.569
Mean weight by outcome.711.568
Confidence rank.720.573

With the readout and calibration objective unchanged, discrepancy weighting gives the highest accuracy in both domains. Averaging weights separately for correct and incorrect answers removes the gain over uniform weighting, even though the total weight assigned to each outcome is preserved. This supports assigning different weights to answers with the same verified outcome. Shuffling confidence scores also lowers accuracy, while replacing them with ranks yields accuracy close to the uniform baseline. CREDO uses the gap between each answer’s confidence and verified correctness to set the strength of its update.


Thank you again for the time and care you have devoted to reviewing our work. Your suggestions have helped us clarify the method and strengthen the experimental comparisons. We would be happy to discuss any remaining questions.

Best regards,
The Authors of Paper 11306

[1] Liu, H., Wang, S., & Xu, H. C²GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning. arXiv:2509.23129, 2025.

[2] Wang, Z., Lou, X., Wu, M., Wen, Z., & Zhang, J. Calibration-Aware Policy Optimization for Reasoning LLMs. ACL 2026.

[3] Chuang, Y.-N., Sarma, P. K., Gopalan, P., Boccio, J., Bolouki, S., Hu, X., & Zhou, H. Learning to Route LLMs with Confidence Tokens. ICML 2025.

[4] Ma, Z., Wen, X., Cao, B., Lu, Y., Lin, H., Yang, J., He, M., Han, X., & Sun, L. Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards. ICML 2026.

[5] Yang, W., Liu, W., Xie, R., Guo, Y., Wu, L., Yang, S., & Lin, Y. LaSeR: Reinforcement Learning with Last-Token Self-Rewarding. ICLR 2026.

Reviewer_hTXYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CREDO, which replaces sampled numerical confidence reports with a differentiable two-token probability readout trained through direct regression within RLVR. It further uses confidence–outcome discrepancies to reweight answer advantages, with experiments demonstrating improved calibration and in-domain accuracy on mathematical and code reasoning benchmarks.

Critical strengths
  1. The paper is well motivated, addressing the important issue of overconfidence in RLVR-trained reasoning models.
  2. The experimental evaluation is thorough, with strong baseline comparisons and informative ablation studies.
  3. The theoretical analysis clearly characterizes the limitations of sampled confidence reports and provides a principled justification for differentiable readouts.
Critical weaknesses
  1. The paper provides limited analysis of why RLCR reduces overconfidence relative to GRPO, particularly whether its lower ECE reflects better-calibrated confidence or a shift toward underconfidence.
  2. All experiments are restricted to Qwen3, leaving generalization to newer generations such as Qwen3.5 and other model families unexplored. If repeating on-policy training across these backbones is prohibitively expensive, evaluating their base-model ECE with verbalized and digit-expectation readouts would still provide a useful reference for whether the diagnosed calibration issues extend beyond Qwen3.
  3. The paper does not compare against simple post-hoc calibration methods, such as temperature scaling applied to digit-expectation confidence scores, leaving unclear how much of the ECE and Brier improvement requires the proposed joint training procedure. Moreover, training uses a fixed rollout temperature without examining its effect on confidence collapse and calibration, even though changing the sampling temperature could affect both the reported confidence distribution and the on-policy training data.
Questions for authors

Please refer to the weakness. Would be happy to raise score once it's well solved.

5 comments
AuthorsOpenReview ↗

Response to Reviewer hTXY (1/3)

Dear Reviewer hTXY,

Thank you for your careful reading and for highlighting our theoretical analysis of sampled confidence reports. Your questions prompted us to examine when confidence is too high or too low, and how well our findings carry over to other models and training conditions.

1. Does RLCR reduce overconfidence or shift the model toward underconfidence?

Weakness 1

The paper provides limited analysis of why RLCR reduces overconfidence relative to GRPO, particularly whether its lower ECE reflects better-calibrated confidence or a shift toward underconfidence.

Thank you for drawing attention to the direction of the error. The base model already tends to overestimate its chances of being right. Under accuracy-only GRPO, an incorrect answer earns the same reward whether the model reports 95% or 30% confidence. The policy can therefore learn to solve more problems while continuing to express high confidence in its mistakes. RLCR changes this incentive through its reward a−(q^−a)2a-(\hat q-a)^2 (Eq. 4), where q^\hat q is the reported confidence and aa is the verified outcome. The calibration term encourages accurate confidence reports by assigning the same penalty to an overestimate and an underestimate of equal size.

Table 1 reports the size of the probability errors (ECE and Brier) and how well confidence distinguishes correct from incorrect answers (AUROC). These measures leave the direction of the error open, so we also report the signed bias, E[c−a]\mathbb E[c-a], which is positive for overconfidence and negative for underconfidence. The table below gives three-seed means for Qwen3-8B, with the standard deviation across seeds shown for signed bias.

DomainMethodSigned biasECE ↓Brier ↓AUROC ↑
MathBase+.468 ± .001.472.453.644
MathGRPO+.189 ± .038.199.204.775
MathRLCR+.107 ± .038.159.171.831
MathDCPO+.215 ± .008.217.198.848
MathCREDO+.004 ± .036.103.086.933
CodeBase+.416 ± .001.418.395.690
CodeGRPO+.030 ± .133.182.205.753
CodeRLCR−.044 ± .007.075.146.815
CodeDCPO−.020 ± .015.153.153.776
CodeCREDO−.009 ± .021.064.127.850

RLCR improves calibration over GRPO in both domains. Its remaining mean bias is toward overconfidence on math and mild underconfidence on code. These gains are consistent with its symmetric calibration reward, which encourages confidence to match verified outcomes without favoring systematically higher or lower reports. CREDO has the smallest absolute mean signed bias, the lowest mean ECE and Brier score, and the highest mean AUROC in both domains, combining more accurate confidence estimates with a closer match to its overall success rate.

2. Does the calibration problem extend beyond Qwen3, and does CREDO transfer?

Weakness 2

All experiments are restricted to Qwen3, leaving generalization to newer generations such as Qwen3.5 and other model families unexplored. If repeating on-policy training across these backbones is prohibitively expensive, evaluating their base-model ECE with verbalized and digit-expectation readouts would still provide a useful reference for whether the diagnosed calibration issues extend beyond Qwen3.

Thank you for suggesting a comparison beyond Qwen3. Table 6 in Appendix C.1 shows that CREDO improves math accuracy and calibration on the 1.7B, 4B, and 8B Qwen3 models. We extended the comparison to Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct to see how these gains transfer to another Qwen generation and to a different model family. (We chose Qwen2.5 because its public weights predate AMC 2024, AIME 2025 and 2026, and our LiveCodeBench v6 subset. Those problems were already public when Qwen3.5 was released, making prior exposure harder to rule out.)

Before RL training. For each answer, we read both the number the model reported and the expected value computed from its digit probabilities (-d). The table gives ECE for the two readouts on the same answers, using evaluation seed 43.

BackboneMath ECE ↓ (verbalized)Math ECE ↓ (digit expectation)Code ECE ↓ (verbalized)Code ECE ↓ (digit expectation)
Qwen3-8B.474.443.418.391
Qwen2.5-7B-Instruct.624.600.559.532
Llama-3.1-8B-Instruct.677.676.567.617

Both Qwen2.5 and Llama show substantial calibration error on math and code under either readout. We next compared how the training methods change answer accuracy and calibration on these same backbones.

AuthorsOpenReview ↗

Response to Reviewer hTXY (2/3)

After RL training. Each cell gives accuracy ↑ / ECE ↓ / AUROC ↑ at seed 43. Within each pair of baseline rows, -d changes only how confidence is read from the same answers.

MethodQwen2.5 · MathQwen2.5 · CodeLlama-3.1 · MathLlama-3.1 · Code
Base.305 / .624 / .608.331 / .559 / .648.138 / .677 / .651.237 / .567 / .614
Base-d.305 / .600 / .628.331 / .532 / .711.138 / .676 / .636.237 / .617 / .647
GRPO.346 / .649 / .513.383 / .394 / .500.189 / .765 / .566.324 / .625 / .500
GRPO-d.346 / .649 / .695.383 / .449 / .749.189 / .778 / .736.324 / .575 / .765
RLCR.329 / .337 / .629.356 / .290 / .730.179 / .151 / .500.308 / .095 / .807
RLCR-d.329 / .275 / .801.356 / .279 / .747.179 / .154 / .519.308 / .102 / .816
DCPO.354 / .263 / .785.367 / .256 / .793.192 / .140 / .773.319 / .204 / .733
DCPO-d.354 / .226 / .856.367 / .260 / .801.192 / .137 / .811.319 / .171 / .795
CREDO.363 / .156 / .884.395 / .233 / .827.203 / .121 / .830.340 / .047 / .839

Using the digit expectation (-d) generally increases baseline AUROC, but does not consistently lower ECE. On both Qwen2.5 and Llama-3.1, CREDO achieves the highest accuracy and AUROC and the lowest ECE among the methods compared. These gains in both math and code show that CREDO’s combination of confidence regression and discrepancy weighting also improves accuracy and calibration beyond Qwen3.

3. How much does post-hoc scaling help, and how does sampling temperature affect confidence?

Weakness 3

The paper does not compare against simple post-hoc calibration methods, such as temperature scaling applied to digit-expectation confidence scores, leaving unclear how much of the ECE and Brier improvement requires the proposed joint training procedure. Moreover, training uses a fixed rollout temperature without examining its effect on confidence collapse and calibration, even though changing the sampling temperature could affect both the reported confidence distribution and the on-policy training data.

3.1 Temperature scaling of digit-expectation scores

Thank you for suggesting this baseline. For each run, we fitted a single temperature to the digit-expectation scores by minimizing log loss on about 20% of the questions. We used the remaining questions to compare the original and scaled scores with CREDO’s native confidence. Arrows show the change after scaling, averaged across three seeds.

MethodMath ECE ↓Math Brier ↓Code ECE ↓Code Brier ↓
Base-d.442 → .193.425 → .251.385 → .205.369 → .249
GRPO-d.182 → .178.196 → .178.153 → .133.179 → .165
RLCR-d.163 → .157.170 → .170.080 → .078.147 → .146
DCPO-d.180 → .143.189 → .174.152 → .121.151 → .143
CREDO (native).103.088.067.125

Temperature scaling improves the baselines, especially the base model. Even without post-hoc temperature scaling, CREDO’s native readout achieves lower mean ECE and Brier scores than every scaled baseline in both domains on the same held-out questions. These results show that post-hoc temperature scaling alone does not close the calibration gap in this comparison, supporting direct supervision of the confidence readout on verified outcomes during RLVR.

AuthorsOpenReview ↗

Response to Reviewer hTXY (3/3)

3.2 Sampling temperature during training and evaluation

Thank you for pointing out that the rollout temperature affects which solutions the policy learns from. We trained RLCR and CREDO once at each temperature (0.7, 1.0, and 1.3) on Qwen3-8B math, keeping evaluation conditions fixed. To test whether warmer rollouts ease confidence collapse, we also report the effective number of confidence levels as defined in Fig. 2.

Training temperatureMethodAcc ↑ECE ↓Brier ↓Effective levels
0.7RLCR.690.158.1563.0
0.7CREDO.754.082.09081.0
1.0RLCR.682.167.1813.8
1.0CREDO.753.094.088111.5
1.3RLCR.683.156.1834.7
1.3CREDO.736.099.110192.4

RLCR’s effective confidence levels increase with training temperature but remain below five, so its confidence collapse persists even with warmer rollouts. CREDO uses far more effective levels and achieves higher accuracy and lower ECE and Brier scores than RLCR at all three training temperatures.

We also vary evaluation sampling temperature for full rollouts, using fixed checkpoints and seed 43. Following the paper’s protocol, we report macro-averaged accuracy and ECE alongside effective confidence levels (Fig. 2).

Evaluation temperatureMethodMath Acc ↑Math ECE ↓Math effective levelsCode Acc ↑Code ECE ↓Code effective levels
0.3Base.445.4571.6.455.4012.6
GRPO.694.1881.6.556.2272.5
RLCR.652.1843.4.517.0747.9
DCPO.690.2391.6.550.1572.1
CREDO.743.10484.1.585.06018.6
0.7Base.440.4741.7.454.4182.6
GRPO.717.1741.7.576.2582.8
RLCR.682.1673.8.539.0708.5
DCPO.715.2221.9.579.1802.0
CREDO.753.094111.5.614.05288.3
1.0Base.433.4781.9.446.4312.8
GRPO.717.1731.7.576.2552.8
RLCR.672.1743.3.535.0698.1
DCPO.709.2251.6.562.1502.1
CREDO.753.091111.8.616.05282.6
1.3Base.401.4772.5.440.4063.4
GRPO.708.1611.8.569.2653.1
RLCR.671.1783.6.540.0718.2
DCPO.711.2221.6.577.1522.1
CREDO.753.102143.2.613.048109.1

Across the tested evaluation temperatures, all four baselines remain below nine effective confidence levels in both domains. This shows that increasing sampling temperature alone cannot overcome the limited resolution of verbalized confidence. CREDO uses more effective levels and achieves the highest accuracy and lowest ECE at every tested temperature in both math and code. Its accuracy and calibration advantages therefore persist beyond the paper’s default evaluation temperature.


Thank you again for the time and care you have devoted to reviewing our work. Your suggestions have helped us clarify the method and strengthen the experimental comparisons. We would be happy to discuss any remaining questions.

Best regards,
The Authors of Paper 11306

Reviewer_nWFhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CREDO, a method for improving confidence calibration during RL training of reasoning models. Instead of having the model write out its confidence as a number in text, the authors compute a continuous score from the output probabilities of two special tokens and train it directly with a regression loss. The answer and an uncertainty-analysis segment are both trained with GRPO, and the gap between the model’s confidence and whether the answer was actually correct is used to adjust how much weight each answer gets during training. On math and code tasks, the paper reports gains in both average accuracy and calibration, and calibration also improves when the model is evaluated on other domains.

Critical strengths

The motivation is clear and ties in well with the method. The paper separates the problems with sampled confidence reports from the problems with reward-only training, which makes a good case for using a differentiable confidence readout.

The main results show gains in both average accuracy and calibration on math and code, and the comparisons are run over three seeds. In the cross-domain setting, calibration also improves while answer accuracy stays about the same.

The analysis of the hybrid target is useful. It lays out both the variance reduction and the bias that the target introduces, and the authors are upfront that matching the mean at the prompt level doesn’t guarantee calibration in general.

Critical weaknesses

W1. It’s not clear what this method offers over simpler ways of getting continuous confidence. The auxiliary-head baseline isn’t a like-for-like comparison, because it uses a different loss, a different target, and a different feedback mechanism. The selective prediction experiments also leave out the digit-expectation (-d) baselines. A comparison against a head trained under matched conditions, plus the -d baselines in the selective prediction results, would make the added value much easier to judge.

W2. The ablations and hyperparameter studies aren’t replicated enough. The main comparisons use three seeds, but most of the component ablations and hyperparameter experiments seem to rely on a single seed. Some of these settings show fairly large accuracy differences, so I’d like to see multi-seed results for at least the key ablations. The authors should also say what validation criterion they used to choose the hyperparameters.

W3. The paper doesn’t show convincingly that better calibration is what drives the accuracy gains. The experiments do show that the weighting helps, but it’s unclear whether the gain comes from the information in the calibrated confidence. On the code task, for example, removing the calibration regression makes ECE much worse while accuracy stays almost the same.

Questions for authors

Please see the Weaknesses section.

I also have one more question. Are the accuracy gains from the uncertainty-analysis segment really due to its uncertainty-specific content, or could they come from a more general benefit of adding an auxiliary RL task? It would help to compare against a different auxiliary analysis task trained with the same budget, which could separate these two explanations.

9 comments
AuthorsOpenReview ↗

Response to Reviewer nWFh (1/5)

Dear Reviewer nWFh,

Thank you for reading our paper so carefully. We appreciate your comments on the motivation for the differentiable readout, the cross-domain results, and the bias–variance analysis of the hybrid target. Your questions led us to extend the readout comparisons and ablations, explain our hyperparameter choices, and examine how confidence learning and uncertainty analysis contribute to accuracy.

1. What does the two-token readout offer over other continuous confidence estimates?

Weakness 1

It’s not clear what this method offers over simpler ways of getting continuous confidence. The auxiliary-head baseline isn’t a like-for-like comparison, because it uses a different loss, a different target, and a different feedback mechanism. The selective prediction experiments also leave out the digit-expectation (-d) baselines. A comparison against a head trained under matched conditions, plus the -d baselines in the selective prediction results, would make the added value much easier to judge.

1.1 A matched scalar head and the choice of token count

Thank you for pointing out what the original head comparison left open. In Appendix C.2, the head used binary cross-entropy to predict answer correctness, while the GRPO policy learned from the accuracy reward alone. Following your suggestion, we matched CREDO’s regression loss, hybrid target, and discrepancy weighting across three alternative readouts.

  • Scalar head. A separate output projection maps the hidden state at the confidence position to a probability of correctness.
  • One reserved token. We apply a sigmoid to the logit of a single learned confidence token.
  • Ten reserved tokens. We assign ten learned tokens to equally spaced confidence levels from 0 to 1 and report the probability-weighted mean of those levels.

The table below extends the readout comparison in Table 8 (Appendix C.2) to these three alternatives, alongside CREDO’s two-token readout on math and code (seed 43).

ReadoutMath Acc ↑Math ECE ↓Math AUROC ↑Code Acc ↑Code ECE ↓Code AUROC ↑
Matched scalar head.669.122.857.563.063.837
One reserved token.684.135.878.552.070.822
Ten reserved tokens.685.113.853.574.072.822
CREDO, two reserved tokens.753.094.933.614.052.869

CREDO’s two-token readout achieves the highest accuracy and AUROC and the lowest ECE in both domains. It uses the model’s vocabulary logits, avoiding the separate output projection needed by the scalar head. With one token, the sigmoid uses zero as a fixed reference. A common shift in all logits therefore changes confidence even though next-token probabilities stay fixed. A ten-token expectation avoids this dependence, but many distributions over the ten levels have the same mean. The regression loss constrains the mean without selecting a unique distribution. CREDO uses the low-confidence token as a learned reference, so a common logit shift cancels in c=pH/(pH+pL)c=p_H/(p_H+p_L). Within this pair, the normalized probabilities are cc and 1−c1-c, so a single regression target specifies their desired split.

AuthorsOpenReview ↗

Response to Reviewer nWFh (2/5)

1.2 Selective prediction with digit-expectation baselines

Thank you for suggesting this comparison. The digit-expectation readout increases AUROC for all four baselines on the same generated answers in both domains (Table 1). We extended the selective-prediction comparison in Table 3 to include both the verbalized and digit-expectation readouts of every baseline, as shown below.

DomainMethodAURC (%) ↓Acc@50 ↑Acc@80 ↑Acc@90 ↑
MathBase29.64 ± 0.30.712 ± .001.705 ± .005.666 ± .001
Base-d22.74 ± 0.24.833 ± .006.716 ± .003.669 ± .002
GRPO7.92 ± 0.67.927 ± .003.915 ± .023.888 ± .011
GRPO-d5.41 ± 0.48.959 ± .013.933 ± .008.898 ± .004
RLCR5.88 ± 0.67.952 ± .009.910 ± .009.878 ± .007
RLCR-d5.68 ± 0.56.955 ± .010.913 ± .008.882 ± .007
DCPO4.98 ± 0.65.958 ± .006.941 ± .002.909 ± .005
DCPO-d5.49 ± 0.95.954 ± .006.940 ± .001.911 ± .006
CREDO2.62 ± 0.04.986 ± .001.962 ± .001.927 ± .002
CodeBase45.10 ± 0.50.572 ± .005.478 ± .002.437 ± .002
Base-d38.33 ± 0.27.639 ± .006.480 ± .002.437 ± .002
GRPO21.50 ± 3.08.797 ± .081.652 ± .041.599 ± .023
GRPO-d20.55 ± 3.10.852 ± .023.664 ± .021.610 ± .010
RLCR21.36 ± 1.00.809 ± .015.608 ± .014.562 ± .014
RLCR-d21.21 ± 0.90.809 ± .017.604 ± .014.559 ± .013
DCPO21.26 ± 0.95.863 ± .016.679 ± .003.611 ± .003
DCPO-d17.53 ± 0.70.877 ± .014.670 ± .012.607 ± .002
CREDO16.80 ± 1.57.885 ± .013.686 ± .025.617 ± .019

Taking the digit expectation removes the noise of sampling a single numeral and generally lowers mean AURC. Even with this improvement, CREDO has the lowest mean AURC and the highest mean accuracy at all three reported coverages on both math and code.

The digit expectation uses probabilities learned under the baseline’s original training objective. The resulting mean can therefore still misestimate how likely an answer is to be correct. RLCR and DCPO still learn confidence through a scalar reward on a sampled numeral, while Base and GRPO receive no calibration supervision (§2.3). CREDO directly supervises the score used for selection against a verifier-derived target and uses it to weight answer updates, helping improve both confidence estimation and answer accuracy.

AuthorsOpenReview ↗

Response to Reviewer nWFh (3/5)

2. Do the component gains hold across seeds, and how were the coefficients chosen?

Weakness 2

The ablations and hyperparameter studies aren’t replicated enough. The main comparisons use three seeds, but most of the component ablations and hyperparameter experiments seem to rely on a single seed. Some of these settings show fairly large accuracy differences, so I’d like to see multi-seed results for at least the key ablations. The authors should also say what validation criterion they used to choose the hyperparameters.

2.1 Three-seed component ablations

We agree that the accuracy differences in Table 2 warrant replication across seeds. We extended these ablations from one training seed to three (43, 44, and 45), with the means and sample standard deviations reported below.

DomainConfigurationAcc ↑ECE ↓AUROC ↑Brier ↓
MathCREDO.752 ± .013.103 ± .013.933 ± .007.086 ± .004
Without calibration regression.713 ± .014.205 ± .047.889 ± .008.195 ± .046
Without uncertainty analysis.712 ± .003.111 ± .014.900 ± .005.109 ± .008
Without policy gradient on the analysis segment.693 ± .007.109 ± .008.900 ± .008.110 ± .008
Without discrepancy weighting.703 ± .025.122 ± .016.912 ± .006.113 ± .014
CodeCREDO.600 ± .018.064 ± .020.850 ± .027.127 ± .016
Without calibration regression.591 ± .022.185 ± .012.834 ± .023.186 ± .013
Without uncertainty analysis.583 ± .012.063 ± .014.847 ± .026.129 ± .013
Without policy gradient on the analysis segment.578 ± .011.067 ± .025.845 ± .020.125 ± .001
Without discrepancy weighting.578 ± .006.058 ± .007.855 ± .011.128 ± .004

Across three seeds, full CREDO achieves the highest mean accuracy in both domains. Removing regression substantially worsens ECE and Brier scores. Mean accuracy also falls when we remove the uncertainty analysis or keep it without its policy-gradient update. This suggests that learning to produce the analysis helps the model answer more accurately. Giving every answer equal weight lowers mean accuracy as well, showing the benefit of using confidence to guide answer updates.

2.2 How we chose the coefficients

We set γ=.5\gamma=.5 to give equal weight to an answer’s verified outcome and the model’s success rate on the same question. With this target fixed, we chose α\alpha and κ\kappa using only short pilot runs on the DeepScaleR and DeepCoder training data. In Weights & Biases, we tracked the readout’s mean squared error against the target and answer accuracy on these training rollouts. We chose the regression weight α=.03\alpha=.03 because it reduced readout error while giving one of the highest accuracies in both domains.

For discrepancy weighting, κ=.25\kappa=.25 and .50.50 gave similar math accuracy in these pilots, so we chose the smaller adjustment, .25.25. On code, .50.50 gave higher accuracy, so we used the stronger weight.

With these coefficients fixed, we trained CREDO independently to completion with seeds 43, 44, and 45 in each domain for the main results in Table 1. We then varied one coefficient at a time in full-length runs with seed 43 (Fig. 4 and Appendix Table 15). These results supported the choices made in the pilot runs.

Reviewer_CAK8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CREDO, a differentiable confidence readout for reinforcement learning with verifiable rewards (RLVR). The authors identify limitations of existing verbalized confidence methods, including sampling variance, confidence collapse, and non-differentiable calibration objectives. CREDO addresses these issues by estimating confidence from the logits of two dedicated tokens and optimizing it through a regression loss. The method also introduces discrepancy-based advantage weighting, which emphasizes rollouts where predicted confidence disagrees with actual correctness. Experiments on mathematical and code reasoning demonstrate improvements in both calibration and accuracy.

Critical strengths
  1. The paper presents a clear motivation and identifies fundamental limitations of existing verbalized confidence approaches. The analysis of sampling variance and non-differentiability provides a convincing justification for the proposed method.

  2. The proposed method is relatively simple and integrates naturally into the existing RLVR framework without requiring a separate confidence prediction model.

  3. The experiments demonstrate consistent improvements in calibration and reasoning accuracy. The paper also includes useful ablations, out-of-domain evaluations, and downstream selective prediction experiments.

Critical weaknesses
  1. Limited understanding of how confidence learning improves reasoning. Although the results demonstrate accuracy improvements, the mechanism through which confidence learning enhances reasoning remains insufficiently explored. In particular, it is unclear whether calibration supervision improves reasoning representations or primarily provides a useful signal for adjusting policy updates. The different effects of removing the calibration loss on mathematical and code reasoning also deserve further investigation.

  2. Limited comparison with alternative uncertainty-based weighting strategies. While the paper evaluates discrepancy weighting through ablation studies, it remains unclear whether calibrated confidence provides unique benefits over simpler uncertainty or difficulty estimates for advantage weighting.

Questions for authors

Q1. Does the confidence calibration objective provide benefits beyond calibration itself?

The paper demonstrates that confidence-related training can improve reasoning accuracy. However, I would appreciate further clarification of the underlying mechanism. Does the differentiable calibration objective directly improve the model's reasoning capabilities, or does it mainly provide more informative signals for discrepancy-based advantage weighting? Additionally, could the authors explain why removing the calibration loss affects mathematical and code reasoning accuracy differently?

Q2. What is the intended role of confidence learning in RLVR relative to exploration?

In conventional RLVR, entropy regularization is often used to maintain exploration and prevent policy collapse. In contrast, I understand that the primary objective of confidence learning in CREDO is calibration, with an additional benefit from discrepancy-based advantage weighting. Is this interpretation correct? Do the authors expect confidence learning or discrepancy weighting to influence exploration and policy diversity, or is their primary purpose to improve calibrated prediction and the allocation of learning signals?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CAK8 (1/3)

Dear Reviewer CAK8,

Thank you for reading our paper so carefully. We appreciate your comments on our analysis of sampled confidence reports and on how naturally the readout fits into RLVR. Your questions prompted us to examine the accuracy gains more closely and compare several ways to weight answer updates. We also discuss how confidence learning relates to exploration.

1. How do calibration regression and discrepancy weighting improve answer accuracy?

Weakness 1

Limited understanding of how confidence learning improves reasoning. Although the results demonstrate accuracy improvements, the mechanism through which confidence learning enhances reasoning remains insufficiently explored. In particular, it is unclear whether calibration supervision improves reasoning representations or primarily provides a useful signal for adjusting policy updates. The different effects of removing the calibration loss on mathematical and code reasoning also deserve further investigation.

Question 1

Does the confidence calibration objective provide benefits beyond calibration itself?

The paper demonstrates that confidence-related training can improve reasoning accuracy. However, I would appreciate further clarification of the underlying mechanism. Does the differentiable calibration objective directly improve the model's reasoning capabilities, or does it mainly provide more informative signals for discrepancy-based advantage weighting? Additionally, could the authors explain why removing the calibration loss affects mathematical and code reasoning accuracy differently?

Thank you for asking about the connection to reasoning accuracy. In CREDO, the same model learns to solve a problem and to estimate how likely its solution is to be correct (Fig. 3). We see two ways this training can help it solve subsequent problems.

  • Learning to report calibrated confidence. We train the two-token readout using both the verifier’s verdict on each answer and the model’s success rate on the same question (Eqs. 6 and 7). The regression loss updates the same parameters used to generate answers, training the solver to judge how likely its own solutions are to be correct. Learning to recognize unreliable reasoning can also help it avoid similar errors on later problems.

  • Learning more from solutions the model misjudged. We use the gap between confidence and verified correctness to weight each answer’s advantage (Eq. 9). A confident mistake reveals a flaw the model overlooked and receives a stronger correction. A correct solution the model doubted receives more reinforcement, helping the policy reproduce a successful approach it had underestimated. Even among answers that earn the same correctness reward, the policy can now give more weight to the ones whose outcomes reveal a mistake in its own judgment.

To examine these effects in both domains, we extended the relevant Table 2 ablations to three training seeds (43, 44, and 45) and added a configuration without either component. The last two rows isolate the effect of adding regression under uniform answer weighting. Entries are means ± sample standard deviations.

Training configurationMath Acc ↑Code Acc ↑
Full CREDO.752 ± .013.600 ± .018
Without calibration regression.713 ± .014.591 ± .022
Without discrepancy weighting.703 ± .025.578 ± .006
Without either component.692 ± .013.562 ± .005

Regression improves mean accuracy even under uniform answer weighting, showing that its benefit extends beyond discrepancy weighting. Using the learned confidence to weight answers improves accuracy further in both domains. Removing regression from full CREDO lowers mean accuracy in both domains, with a larger drop on math.

AuthorsOpenReview ↗

Response to Reviewer CAK8 (2/3)

2. How does discrepancy weighting compare with weighting by uncertainty or question difficulty?

Weakness 2

Limited comparison with alternative uncertainty-based weighting strategies. While the paper evaluates discrepancy weighting through ablation studies, it remains unclear whether calibrated confidence provides unique benefits over simpler uncertainty or difficulty estimates for advantage weighting.

Thank you for suggesting these comparisons. Uncertainty measures how unsure the model is, and question difficulty measures how often it fails on a question. Discrepancy instead measures how far each answer’s confidence is from its verified outcome.

With the same confidence readout and calibration objective, we compared six alternative ways to weight the answer advantage on math and code (seed 43). Three use simpler signals, and three modify the information that enters the discrepancy weight.

  1. Uncertainty. Use 4ci(1−ci)4c_i(1-c_i), which is highest when the model assigns equal probability to success and failure.
  2. Question difficulty. Use each question’s error rate, centered across questions in the rollout batch, to assign the same weight to all its answers.
  3. Errors only. Score an incorrect answer as 1 and a correct answer as 0.
  4. Shuffled confidence. Shuffle confidence scores among answers to the same question before computing each answer’s discrepancy.
  5. Mean weight by outcome. Within each question, average the CREDO weights separately for correct and incorrect answers. Give each answer the mean for its group.
  6. Confidence rank. Replace each confidence score with its percentile rank among answers to the same question before computing the discrepancy.
Answer weightingMath Acc ↑Code Acc ↑
CREDO discrepancy.753.614
Uniform.721.572
Uncertainty.730.599
Question difficulty.739.601
Errors only.706.583
Shuffled confidence.674.569
Mean weight by outcome.711.568
Confidence rank.720.573

Weighting by uncertainty or question difficulty improves accuracy over uniform weighting, but CREDO achieves the highest accuracy in both domains. Shuffling confidence, averaging weights by outcome, or replacing confidence with ranks all reduce accuracy relative to CREDO. These comparisons support using each answer’s own confidence value together with its verified outcome to determine the strength of its update.

To see what these rules would prioritize in practice, we examined two solutions sampled for the same MATH-500 test problem. Both used inclusion–exclusion, but one made an arithmetic error. The table gives their confidence scores and the corresponding weighting signals.

Sampled answerConfidence ccUncertainty 4c(1−c)4c(1-c)Discrepancy ∣a−c∣\lvert a-c\rvert
3 (correct).958.161.042
1 (incorrect).967.128.967

Uncertainty gives the correct solution the higher score because the model was slightly less sure of it. Discrepancy would put more weight on the solution with the arithmetic error. Giving such answers a stronger corrective update helps the model learn to avoid mistakes it still overlooks when judging its own work.

AuthorsOpenReview ↗

Response to Reviewer CAK8 (3/3)

3. How does confidence learning relate to exploration and policy diversity?

Question 2

What is the intended role of confidence learning in RLVR relative to exploration?

In conventional RLVR, entropy regularization is often used to maintain exploration and prevent policy collapse. In contrast, I understand that the primary objective of confidence learning in CREDO is calibration, with an additional benefit from discrepancy-based advantage weighting. Is this interpretation correct? Do the authors expect confidence learning or discrepancy weighting to influence exploration and policy diversity, or is their primary purpose to improve calibrated prediction and the allocation of learning signals?

Thank you for drawing this distinction. Yes, your interpretation captures our intent. As discussed in Responses 1 and 2, learning to assess its own reasoning helps the model report calibrated confidence and can also improve the solutions it produces. Using that confidence in the answer update gives the policy a stronger signal on solutions it misjudged, improving accuracy further.

Our primary goals are therefore calibrated prediction and better allocation of the learning signal. CREDO improves both calibration and accuracy over GRPO on the in-domain math and code benchmarks (Table 1).

We expect the effect on exploration to be indirect. As confidence estimates improve, discrepancy weighting can change which solution strategies the policy reinforces. If it reinforces a familiar successful approach, future answers may become more concentrated. If it reinforces successful approaches the model rarely uses, future solutions may become more varied. How diversity changes therefore depends on which approaches are reinforced, whereas entropy regularization explicitly encourages a broader next-token distribution.


Thank you again for the time and care you have devoted to reviewing our work. Your suggestions have helped us clarify the method and strengthen the experimental comparisons. We would be happy to discuss any remaining questions.

Best regards,
The Authors of Paper 11306

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary with Thanks to the Reviewers and Chairs

Dear ICLR 2027 Reviewers and Chairs,

As the discussion draws to a close, we thank the reviewers for their careful reading of our paper and the chairs for overseeing the review process. We briefly summarize our contribution and the experiments prompted by the reviews below.

We show that sampling confidence as text in RLVR adds noise and blocks gradients through the reported score. CREDO trains a two-token confidence readout by regression and uses the gap between confidence and correctness to weight answer updates. CREDO achieves higher answer accuracy and lower calibration error on mathematical reasoning and code generation benchmarks, and its confidence scores improve selective prediction.

We appreciate the reviewers’ comments on the following strengths of the work.

  • Reviewers CAK8, nWFh, and hTXY: The analysis identifies the limitations of sampled confidence reports and gives a clear motivation for a differentiable readout.
  • Reviewers CAK8 and ibnH: Reading confidence from the probabilities of two tokens provides a simple way to estimate how likely the model’s answer is to be correct.
  • Reviewers CAK8 and kKWR: The confidence readout fits naturally into RLVR, allowing the reasoning policy to learn to assess its own answers during training.
  • Reviewers CAK8, nWFh, and ibnH: The experiments show gains in both answer accuracy and calibration across mathematical reasoning and programming tasks.
  • Reviewers hTXY, kKWR, and ibnH: The evaluation includes strong baselines and informative component ablations that help assess which training choices contribute to CREDO’s gains.
  • Reviewers CAK8, kKWR, and ibnH: The selective prediction experiments test how well the model’s confidence scores identify reliable answers when it can abstain.

During the rebuttal, we responded to the questions raised in all five reviews. We summarize the main clarifications and additional results below.

  • Comparison with other confidence methods. CREDO achieved higher accuracy and lower calibration error than a scalar head trained under matched conditions and four additional baselines, including Self-REF and LaSeR.
  • Contribution of calibration regression. Across three seeds, regression improved mean answer accuracy even when all answers received equal weight.
  • Contribution of discrepancy weighting. With the readout held fixed, discrepancy weighting gave higher answer accuracy than weighting by uncertainty, weighting by question difficulty, or LaSeR-style advantage mixing.
  • Role of uncertainty analysis. Under the same training budget, analyzing uncertainty improved accuracy more than summarizing the solution. At inference, removing the analysis worsened calibration even when the answer was held fixed.
  • Selective prediction. After adding digit-expectation baselines, CREDO retained the highest mean accuracy at all three coverage levels we evaluated.
  • Direction of calibration error. On average, RLCR remained overconfident on mathematical reasoning tasks and mildly underconfident on programming tasks.
  • Temperature controls. Post-hoc scaling improved baseline calibration without closing the gap to CREDO. At higher sampling temperatures, RLCR’s reported confidence still clustered around a few values.
  • Generalization. Gains in accuracy and calibration held on Qwen2.5 and Llama-3.1. In a separate transfer evaluation across eight additional datasets, both the math-trained and code-trained CREDO models achieved higher average accuracy and lower average calibration error than the baselines.

Thank you again for the time and care you put into reviewing our paper. Your feedback helped us explain our approach more clearly and evaluate it more thoroughly.

Best regards,
The Authors of Paper 11306

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.