- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CREDO, a method for training calibrated reasoning models. Instead of sampling verbalized confidence scores, CREDO uses a differentiable readout based on the relative probabilities of two dedicated tokens. Experiments on mathematical and code reasoning demonstrate improvements in both accuracy and calibration over standard GRPO and calibration baselines such as RLCR, and DCPO.
Critical strengths
-
Simple and effective approach: Verbalized confidence suffers from issues such as entropy collapse and variance. The differentiable readout is a natural alternative to verbalized confidence which fixes some of these issues.
-
Strong empirical results: CREDO achieves substantial improvements in calibration, while interestingly also improving accuracy over standard RLVR.
-
Thorough experimental evaluation: The paper includes strong baselines, multiple ablations, different model scales, and downstream evaluations of selective prediction. I particularly appreciate the experiments disentangling the confidence readout from the training objective.
Critical weaknesses
-
Understanding the accuracy improvements. One of the most interesting findings is that CREDO substantially improves accuracy over vanilla GRPO, whereas other calibration methods (RLCR and DCPO) do not show similarly consistent gains. While the ablations identify contributing components, they do not fully explain why these improvements occur. If there is a synergy between calibration and correctness optimization, why don't other methods benefit similarly? Investigating the underlying cause of this would be useful- it could be because of potential differences in reasoning length, reward design (particularly the format reward), and training dynamics.
-
Novelty relative to Chuang et al. (2025). The confidence-token readout is adopted from [1]. The primary contributions therefore appear to be incorporating this mechanism into RLVR and introducing discrepancy weighting. A clearer discussion of the differences and what specifically enables the improvements in this setting, would strengthen the paper.
-
Limited OOD evaluation. The OOD experiments are restricted to transfer between mathematics and code. Given that evaluating additional datasets requires no further training, it would be nice to see a broad range of both reasoning and non-reasoning tasks in this evaluation.
[1]: Chuang, Y. N., Sarma, P. K., Gopalan, P., Boccio, J., Bolouki, S., Hu, X., & Zhou, H. (2024). Learning to route llms with confidence tokens. arXiv preprint arXiv:2410.13284.
Questions for authors
-
Accuracy gains: What do the authors believe drives the substantial accuracy improvements over vanilla GRPO? Why don't other calibration methods such as RLCR and DCPO benefit similarly? Could differences in reasoning length, format rewards, or training dynamics explain these gains?
-
Confidence reasoning: Does the differentiable confidence readout actually rely on the verbalized uncertainty analysis? Interestingly, removing this analysis substantially hurts accuracy while having relatively little effect on calibration, particularly for code. Since the analysis is generated after the answer, the accuracy gains must arise from training dynamics rather than a direct effect on the generated answer. It would be interesting to conduct an inference-time intervention where the confidence analysis is removed or replaced with an unrelated analysis, while keeping the model and answer fixed, to understand whether the content of the analysis meaningfully contributes to confidence estimation.
-
Comparison with Chuang et al.: see weaknesses
-
OOD Eval: see weaknesses
-
Quality of confidence reasoning: Are the generated confidence analysis better than baselines? Some qualitative evaluations of this would be helpful - for example comparing the confidence analysis of RLVR (should be bad), RLCR, etc to Credo.
-
Reasoning length: How do the average reasoning lengths compare across CREDO, GRPO, RLCR, and DCPO?
7 comments
Response to Reviewer ibnH (2/5)
A second difference is that neither original baseline weights an answer’s advantage by the gap between its confidence and verified correctness. We therefore retrained RLCR and DCPO with discrepancy weighting, using either the reported number or the digit expectation (Appendix C.3, Table 9).
| Source of in the discrepancy | RLCR math Acc ↑ | RLCR code Acc ↑ | DCPO math Acc ↑ | DCPO code Acc ↑ |
|---|---|---|---|---|
| Original objective (no discrepancy weighting) | .682 | .539 | .715 | .579 |
| Reported number | .701 | .545 | .716 | .581 |
| Digit expectation | .697 | .542 | .690 | .560 |
Discrepancy weighting improves RLCR’s accuracy, especially on math. For DCPO it makes little difference with the reported number and lowers accuracy in both domains with the digit expectation. These uneven gains point to the quality of the confidence signal. When a score poorly estimates correctness, its distance from the verified outcome is a less reliable guide to which answers deserve a stronger update. Direct regression gives CREDO a better basis for identifying confident errors and underestimated successes. Using that distinction to weight the answer updates improves accuracy in both domains.
1.2 Could answer length, format rewards, or training dynamics explain the gain?
Question 6
Reasoning length: How do the average reasoning lengths compare across CREDO, GRPO, RLCR, and DCPO?
Answer length.
| Method | Math answer tokens | Code answer tokens |
|---|---|---|
| GRPO | 3,403.3 ± 314.3 | 2,874.9 ± 264.2 |
| RLCR | 2,762.6 ± 187.2 | 1,648.9 ± 157.1 |
| DCPO | 3,557.3 ± 344.4 | 2,853.3 ± 405.7 |
| CREDO | 3,514.6 ± 140.5 | 2,354.0 ± 267.1 |
CREDO’s math answers are slightly longer on average than GRPO’s and shorter than DCPO’s. On code, CREDO achieves higher mean accuracy with shorter answers than either GRPO or DCPO. The accuracy gains therefore do not require consistently longer answers.
Format reward. We compared three ways of rewarding the answer in CREDO (Appendix C.4, Table 10). Here indicates a correct answer, indicates valid formatting, and is the format bonus.
| Answer reward | Math Acc ↑ | Code Acc ↑ | |
|---|---|---|---|
| Correctness only | .723 | .583 | |
| Format-gated reward | .714 | .589 | |
| Separate format bonus (CREDO) | .753 | .614 |
When CREDO and GRPO both receive only a correctness reward for the answer, CREDO still achieves higher accuracy in both domains. Its advantage over GRPO therefore persists without a format bonus. We use the separate bonus because it gives the highest accuracy among the three reward designs. It preserves the full correctness reward when formatting fails, while still encouraging the requested format.
Training dynamics. We tracked how accurately the model predicted its calibration target throughout training. The table gives the mean squared error against that target in early, middle, and late training, averaged over three seeds.
| Training domain | Configuration | Early | Middle | Late |
|---|---|---|---|---|
| Math | Full CREDO | .296 | .084 | .073 |
| Math | Without calibration regression | .301 | .118 | .115 |
| Math | Without discrepancy weighting | .294 | .090 | .080 |
| Code | Full CREDO | .350 | .101 | .096 |
| Code | Without calibration regression | .394 | .186 | .176 |
| Code | Without discrepancy weighting | .346 | .110 | .102 |
By mid-training, both configurations with regression predict their targets more accurately than the variant trained without it, and the difference persists through the final stage in both domains. Full CREDO and the variant with uniform weights finish with similar prediction errors on their own solutions. What distinguishes them is how they use those estimates. The policy that gives extra weight to answers it misjudged achieves higher held-out accuracy in both domains (Response 1.1).
Response to Reviewer ibnH (3/5)
2. What does CREDO add to the confidence tokens of Chuang et al.?
Weakness 2
Novelty relative to Chuang et al. (2025). The confidence-token readout is adopted from [1]. The primary contributions therefore appear to be incorporating this mechanism into RLVR and introducing discrepancy weighting. A clearer discussion of the differences and what specifically enables the improvements in this setting, would strengthen the paper.
Question 3
Comparison with Chuang et al.: see weaknesses
Thank you for highlighting this connection. Chuang et al. [1] introduced the two-token confidence readout we use. Self-REF learns it through supervised fine-tuning to decide when to accept an answer and when to route the question elsewhere. That objective limits what the readout can teach the solver. Its supervision comes from a fixed set of answers produced by the original model, and on a wrong answer the loss masks the answer tokens and trains the uncertainty token. The error teaches the system when to defer, but the confidence score is not used to strengthen a corrective update to the incorrect answer.
In RLVR the policy keeps changing, along with the errors it needs to recognize. CREDO therefore trains the readout on each new batch of verified solutions and uses the gap between confidence and correctness to weight each answer’s policy update. A confidently wrong answer can now teach the solver both to revise its judgment and to avoid repeating the mistake.
We compared Self-REF and CREDO on Qwen3-8B across three training seeds.
| Training domain | Method | Acc ↑ | ECE ↓ | Brier ↓ | AUROC ↑ |
|---|---|---|---|---|---|
| Math | Self-REF | .457 ± .006 | .298 ± .095 | .318 ± .084 | .528 ± .049 |
| Math | CREDO | .752 ± .013 | .103 ± .013 | .086 ± .004 | .933 ± .007 |
| Code | Self-REF | .450 ± .001 | .385 ± .013 | .367 ± .010 | .793 ± .017 |
| Code | CREDO | .600 ± .018 | .064 ± .020 | .127 ± .016 | .850 ± .027 |
With the same two-token readout, CREDO achieves higher mean accuracy and AUROC and lower mean ECE and Brier scores than Self-REF in both domains. These gains support CREDO’s approach of training the readout on the policy’s own verified rollouts and using confidence errors to guide answer updates.
3. Does calibration transfer beyond mathematics and code?
Weakness 3
Limited OOD evaluation. The OOD experiments are restricted to transfer between mathematics and code. Given that evaluating additional datasets requires no further training, it would be nice to see a broad range of both reasoning and non-reasoning tasks in this evaluation.
Question 4
OOD Eval: see weaknesses
Thank you for suggesting a broader test of transfer. We report results from the existing math-trained and code-trained checkpoints on eight datasets covering factual QA, reading comprehension, commonsense and logical reasoning, general knowledge, and biomedical QA. Each cell reports single-checkpoint accuracy ↑ / ECE ↓ / Brier ↓ / AUROC ↑. The macro average weights the eight datasets equally. Bold marks the best displayed value for each metric within each row, including ties.
Math-trained models
| Dataset | Base | GRPO | RLCR | DCPO | CREDO |
|---|---|---|---|---|---|
| NQ-Open | .186/.757/.728/.654 | .210/.723/.688/.642 | .203/.716/.683/.654 | .210/.719/.707/.618 | .212/.311/.259/.740 |
| BoolQ | .466/.492/.480/.619 | .625/.323/.326/.658 | .616/.328/.326/.696 | .621/.360/.351/.653 | .799/.190/.149/.830 |
| StrategyQA | .395/.520/.487/.638 | .500/.405/.384/.708 | .546/.338/.321/.728 | .522/.405/.370/.696 | .535/.160/.192/.821 |
| LogiQA 2.0 | .638/.303/.318/.562 | .731/.195/.224/.587 | .738/.176/.221/.553 | .732/.240/.244/.660 | .732/.144/.193/.699 |
| MMLU-Pro | .498/.448/.444/.617 | .615/.277/.287/.700 | .622/.279/.289/.713 | .641/.304/.303/.747 | .641/.113/.178/.818 |
| MedQA | .574/.373/.381/.553 | .689/.250/.272/.567 | .696/.225/.255/.602 | .701/.268/.272/.651 | .720/.167/.194/.758 |
| MedMCQA | .474/.477/.473/.549 | .562/.378/.386/.557 | .595/.337/.347/.591 | .614/.357/.357/.613 | .620/.078/.204/.740 |
| PubMedQA | .552/.356/.349/.696 | .542/.358/.358/.664 | .561/.320/.326/.678 | .584/.366/.355/.690 | .627/.163/.217/.738 |
| Macro average | .473/.466/.458/.611 | .559/.364/.366/.635 | .572/.340/.346/.652 | .578/.377/.370/.666 | .611/.166/.198/.768 |
Response to Reviewer ibnH (1/5)
Dear Reviewer ibnH,
Thank you for your careful reading and for highlighting the experiments that separate the confidence readout from its training objective. Your observation that the analysis comes after the answer helped us distinguish what the model learns for future problems from what it uses to judge an answer it has already given. Below, we address the accuracy gains, the comparison with Self-REF, and transfer beyond math and code. We then examine the uncertainty analyses through fixed-answer interventions and qualitative examples.
1. Why does CREDO improve answer accuracy over GRPO?
1.1 How learning and using confidence contribute to accuracy
Thank you for asking why CREDO improves accuracy more consistently than RLCR and DCPO. In CREDO, the model that solves a problem also learns to assess its solution. Its confidence contributes to learning in two ways (Fig. 3).
We extended the two individual ablations in Table 2 to three training seeds. We also added a fourth configuration that switches off both regression and discrepancy weighting, so we can measure the effect of regression under uniform answer weights as well.
Regression improves calibration and mean accuracy in both domains even under uniform answer weighting. Using the learned confidence to weight answers adds a further gain, giving full CREDO the highest mean accuracy among the four configurations.
Why are the gains less consistent for RLCR and DCPO? RLCR [2] rewards the confidence report together with the answer, while DCPO [3] rewards it separately (§2.2). Both sample a number, so variation across reports adds to the expected squared calibration error even for a fixed solution and analysis. Because the sampled numeral is discrete, calibration feedback reaches the model through a policy-gradient update based on its reward. CREDO trains its continuous score directly by regression, giving the shared solver feedback on the size and direction of its confidence error (§2.3, Eq. 5).
Taking the digit expectation at evaluation improves baseline AUROC, but CREDO still achieves better calibration in both domains (Table 1). Training still relies on rewards for sampled reports, leaving the continuous score without direct supervision. We think this indirect feedback makes it harder for the solver to anticipate its own reasoning errors, so confidence training brings less consistent accuracy gains.