- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies reward function design for jointly improving answer correctness and verbalized confidence calibration in large language models through reinforcement learning. The authors formalize non-hackable confidence reward schemes using three properties: interpretability, proper scoring, and best effort. They provide a characterization of these reward schemes and establish connections with existing abstention-based reward frameworks. Furthermore, they introduce a spectrum of reward schemes with asymmetric penalties for overconfidence and underconfidence. Experiments on mathematical reasoning and question-answering datasets demonstrate that improperly designed reward schemes may lead to substantial accuracy degradation and increased abstention, especially on difficult questions. The authors also investigate how different non-hackable reward schemes affect empirical confidence calibration.
Critical strengths
- The paper gives a useful characterization of the additional condition needed beyond proper scoring. In the stated smooth binary setting, the derivative relations connect optimal confidence reporting to the monotonicity of the best achievable expected reward. The connection to weighted abstention-threshold schemes makes the result constructive and places multiple existing approaches in one framework (Section 3 and Appendix A).
- The experiments demonstrate a consequential failure mode rather than only small improvements in calibration metrics. For Qwen2.5-3B on hard HotpotQA-Modified examples, accuracy falls from 0.2100 under correctness-only rewards to 0.0204 under Log-1 and 0.0958 under Brier-2 (Table 2). The additional abstention analyses and calibration metrics help connect the theoretical incentives to observed behavior.
Critical weaknesses
- Insufficient evaluation of training stability and reproducibility. The experimental results are primarily based on single training runs, without reporting means and standard deviations across multiple random seeds. Given the inherent instability of RL training, it is difficult to determine whether the observed differences between reward schemes are statistically reliable. Moreover, the authors acknowledge training instability and unsuccessful runs in Appendix I, which further raises concerns about the robustness of the reported results. I encourage the authors to conduct multi-seed experiments and report the mean performance, standard deviation, and training failure rate for each reward scheme.
- Theoretical non-hackability does not necessarily translate into accuracy preservation in practice. Although the proposed non-hackability conditions ensure that higher correctness probabilities lead to higher optimal expected rewards, some strictly non-hackable reward schemes still result in substantial accuracy degradation after RL training. For instance, on DeepMath-103K with Qwen2.5-3B, Brier-log Hybrid achieves only 26.35% accuracy, compared with 42.38% for the correctness-only baseline. This suggests a considerable gap between the theoretical incentive guarantees and practical RL optimization outcomes. The authors should investigate the underlying causes of such degradation and clarify the extent to which the proposed theoretical framework can reliably guide reward function selection in practice.
Questions for authors
- Do the key comparisons and spectrum trends persist across independent training seeds and when invalid or truncated outputs are separated from valid-answer errors? A small representative subset would already be informative.
- How much of the spectrum's empirical ordering remains after accounting for reward variance or effective update size? Please distinguish the proven penalty asymmetry from the hypothesized effect of finite optimization.
- For HotpotQA-Modified examples with removed evidence, how do the authors distinguish reward-driven avoidable abstention from appropriate uncertainty under missing information? Please scope the claim to the evaluation convention that abstention counts as incorrect.
5 comments
Reviewer GPQ2 Rebuttal (Part 2)
W2: Non-hackable confidence theoretically may not imply non-hackable confidence practically due to optimization dynamics.
Theoretical non-hackability does not necessarily translate into accuracy preservation in practice. Although the proposed non-hackability conditions ensure that higher correctness probabilities lead to higher optimal expected rewards, some strictly non-hackable reward schemes still result in substantial accuracy degradation after RL training. For instance, on DeepMath-103K with Qwen2.5-3B, Brier-log Hybrid achieves only 26.35% accuracy, compared with 42.38% for the correctness-only baseline. This suggests a considerable gap between the theoretical incentive guarantees and practical RL optimization outcomes. The authors should investigate the underlying causes of such degradation and clarify the extent to which the proposed theoretical framework can reliably guide reward function selection in practice.
The reason is likely due to the gradient conflict between accuracy and confidence calibration when they are jointly optimized, leading to a trade-off in accuracy and confidence calibration [1]. Each iteration, the LLM can only priortize accuracy, or confidence calibration, but not both. The overconfidence penalty is more severe relative to the underconfidence penalty gradient for non-hackable confidence reward schemes nearer to the underconfident end of the spectrum. This implies that the expected reward is likely higher prioritizing confidence calibration over accuracy nearer to the underconfident end of the spectrum. Therefore, reward schemes in the underconfident end of the spectrum prioritize confidence calibration, while reward schemes nearer to the overconfident end of the spectrum prioritize accuracy. We have since updated our paper to include this phenomenon as a limitation of our work, and included a more detailed and intuitive explanation of this phenomenon to Appendix H.2.
W3: Effect of effective batch size on overconfidence to underconfidence spectrum ordering
How much of the spectrum's empirical ordering remains after accounting for reward variance or effective update size? Please distinguish the proven penalty asymmetry from the hypothesized effect of finite optimization.
The trends in the overconfidence to underconfidence spectrum hold up to expected statistical noise of up to approximately 0.1, based on standard deviations mostly in the order of 0.01 to 0.1 when repeating Gemma 4 (E2B) Instruct three times. We have run experiments on Gemma 4 (E2B) Instruct using HotpotQA-Modified dataset. We have doubled the number of responses per question to 16 in one experiment, and doubled the number of questions per batch to 128 in a separate experiment. The results are as follows:
Doubled the number of responses per question to 16
| Model/Reward Scheme | Accuracy | AUROC | Brier-1 | Brier | ECE | Calib. Bias |
|---|---|---|---|---|---|---|
| Log Loss | 0.0000 | - | -0.0001 | 0.0001 | 0.0050 | -0.0050 |
| Brier Score | 0.0000 | - | 0.0000 | 0.0000 | 0.0050 | -0.0050 |
| Log-1 | 0.0000 | - | 0.0000 | 0.0000 | 0.0050 | -0.0050 |
| Brier-2 | 0.0007 | - | -0.0009 | 0.0016 | 0.0065 | -0.0065 |
| Log- | 0.4322 | 0.7536 | 0.2254 | 0.2068 | 0.0885 | 0.0274 |
| Correctness-only | 0.3980 | 0.5057 | -0.1944 | 0.5924 | 0.5939 | -0.5915 |
| Overconfidence-1000 | 0.4572 | 0.7497 | -0.0024 | 0.4595 | 0.4846 | -0.4846 |
| Overconfidence-4 | 0.4424 | 0.7734 | 0.1960 | 0.2464 | 0.1975 | -0.1933 |
| Overconfidence-1 | 0.4522 | 0.7993 | 0.2551 | 0.1971 | 0.1132 | -0.0379 |
| Brier-1 | 0.4507 | 0.7650 | 0.2272 | 0.2235 | 0.1647 | -0.0202 |
| Underconfidence-1 | 0.4566 | 0.7974 | 0.2712 | 0.1854 | 0.0557 | -0.0125 |
| Underconfidence-4 | 0.4249 | 0.7585 | 0.2269 | 0.1980 | 0.0516 | -0.0470 |
| Brier-log Hybrid | 0.4278 | 0.7707 | 0.2156 | 0.2122 | 0.1308 | -0.0512 |
(to be continued)
Reviewer GPQ2 Rebuttal (Part 3)
W3 (continued)
Doubled the number of questions per batch to 128
| Model/Reward Scheme | Accuracy | AUROC | Brier-1 | Brier | ECE | Calib. Bias |
|---|---|---|---|---|---|---|
| Log Loss | 0.0000 | - | -0.0008 | 0.0008 | 0.0057 | -0.0057 |
| Brier Score | 0.0000 | - | 0.0000 | 0.0000 | 0.0050 | -0.0050 |
| Log-1 | 0.0105 | 0.9437 | 0.0067 | 0.0038 | 0.0057 | -0.0056 |
| Brier-2 | 0.3136 | 0.8942 | 0.1931 | 0.1205 | 0.0772 | -0.0626 |
| Log- | 0.4566 | 0.7494 | 0.2406 | 0.2159 | 0.1146 | 0.0055 |
| Correctness-only | 0.4141 | 0.5000 | -0.1660 | 0.5801 | 0.5809 | -0.5809 |
| Overconfidence-1000 | 0.3279 | 0.5001 | -0.3375 | 0.6654 | 0.6671 | -0.6671 |
| Overconfidence-4 | 0.4590 | 0.7774 | 0.1991 | 0.2599 | 0.2310 | -0.1796 |
| Overconfidence-1 | 0.4440 | 0.7745 | 0.2222 | 0.2218 | 0.1362 | -0.1262 |
| Brier-1 | 0.4337 | 0.7587 | 0.2198 | 0.2139 | 0.1345 | -0.0332 |
| Underconfidence-1 | 0.4493 | 0.7813 | 0.2556 | 0.1938 | 0.0663 | -0.0196 |
| Underconfidence-4 | 0.4567 | 0.7772 | 0.2550 | 0.2017 | 0.0952 | -0.0028 |
| Brier-log Hybrid | 0.4232 | 0.7421 | 0.2174 | 0.2058 | 0.0834 | -0.0746 |
W4: The abstention claim should come with a caveat mentioning that abstentions are considered wrong answers.
For HotpotQA-Modified examples with removed evidence, how do the authors distinguish reward-driven avoidable abstention from appropriate uncertainty under missing information? Please scope the claim to the evaluation convention that abstention counts as incorrect.
Thank you for your suggestion. Assuming abstentions are considered wrong answers, all reward-driven abstention is avoidable due to the existence of non-hackable confidence reward schemes. We have updated the second paragraph of Section 3.1 to make it explicit that in our problem setting, abstentions are considered incorrect. In particular, abstention in our setting is avoidable due to the existence of non-hackable confidence reward schemes, therefore, the ideal rate of abstention is zero, and all abstention is reward-driven. However, due to various factors such as the gradient conflict between accuracy and confidence when they are jointly optimized [1], abstentions can exist even in non-hackable confidence reward schemes as LLMs prioritize confidence optimization to varying degrees. A more detailed explanation can be found in Appendix H.2.
References:
[1] Zhengzhao Ma, Xueru Wen, Boxi Cao, Yaojie Lu, Hongyu Lin, Jinglin Yang, Min He, Xianpei Han, and Le Sun. Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=3V1p2bJugq.
Reviewer GPQ2 Rebuttal (Part 1)
W1: 'Insufficient evaluation of training stability and reproducibility'
We thank you for your suggestion. We have repeated our experiments for Gemma 4 (E2B) Instruct three times. The exception is Brier-log Hybrid where the third attempt is on the process of evaluation. Results will be updated upon acceptance. The formatting reward is set to 0.5. To discourage overly long responses, an overlong penalty of 0 for response lengths at most 2048 tokens, 0.5 for response lengths at the maximum length of 4096 tokens, and linearly interpolated between 2048 and 4096 tokens, is set. We empirically found that this configuration is more stable, and less prone to training collapse without resorting to large formatting reward.
The criteria for training collapse are at least one of the following:
Of these 39 RL experiments, only 3 resulted in LLM training collapse. We have reported which reward schemes have collapsed in Appendix F for the main experiments and Appendix I for the ablation studies.
We have since updated Appendix F to include the new results. For ease of reference, the results of the repeated Gemma 4 (E2B) Instruct experiments are as follows, in the format of mean ± standard deviation:
(to be continued)