acceptodds
Under review as a conference paper at ICLR 2027

On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models

Abstract

In this paper, we consider the setting where large language models (LLMs) are trained using reinforcement learning (RL) to simultaneously improve reasoning accuracy and verbalize their confidence. Our reward scheme uses two functions for rewarding confidence verbalized by the LLM: one for correct answers and the other for incorrect answers. If poorly designed, such a scheme may incentivize an LLM to answer incorrectly in order for its confidence to be calibrated, a phenomenon we term confidence reward hacking. We introduce the notion of non-hackable confidence reward schemes and provide methods for constructing them. We show that selective confidence reward hacking can arise in practical datasets under hackable reward schemes while non-hackable reward schemes are resistant to hacking. Finally, we place some of these schemes along an overconfidence–underconfidence spectrum for RL-based confidence calibration and demonstrate experimentally that they tend to exhibit the corresponding calibration biases relative to other schemes in the spectrum. The code of our experiments is available in https://anonymous.4open.science/r/rl-confidence-calibration-9ED4/README.md.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reward hacking, confidence calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GPQ2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reward function design for jointly improving answer correctness and verbalized confidence calibration in large language models through reinforcement learning. The authors formalize non-hackable confidence reward schemes using three properties: interpretability, proper scoring, and best effort. They provide a characterization of these reward schemes and establish connections with existing abstention-based reward frameworks. Furthermore, they introduce a spectrum of reward schemes with asymmetric penalties for overconfidence and underconfidence. Experiments on mathematical reasoning and question-answering datasets demonstrate that improperly designed reward schemes may lead to substantial accuracy degradation and increased abstention, especially on difficult questions. The authors also investigate how different non-hackable reward schemes affect empirical confidence calibration.

Critical strengths
  1. The paper gives a useful characterization of the additional condition needed beyond proper scoring. In the stated smooth binary setting, the derivative relations connect optimal confidence reporting to the monotonicity of the best achievable expected reward. The connection to weighted abstention-threshold schemes makes the result constructive and places multiple existing approaches in one framework (Section 3 and Appendix A).
  2. The experiments demonstrate a consequential failure mode rather than only small improvements in calibration metrics. For Qwen2.5-3B on hard HotpotQA-Modified examples, accuracy falls from 0.2100 under correctness-only rewards to 0.0204 under Log-1 and 0.0958 under Brier-2 (Table 2). The additional abstention analyses and calibration metrics help connect the theoretical incentives to observed behavior.
Critical weaknesses
  1. Insufficient evaluation of training stability and reproducibility. The experimental results are primarily based on single training runs, without reporting means and standard deviations across multiple random seeds. Given the inherent instability of RL training, it is difficult to determine whether the observed differences between reward schemes are statistically reliable. Moreover, the authors acknowledge training instability and unsuccessful runs in Appendix I, which further raises concerns about the robustness of the reported results. I encourage the authors to conduct multi-seed experiments and report the mean performance, standard deviation, and training failure rate for each reward scheme.
  2. Theoretical non-hackability does not necessarily translate into accuracy preservation in practice. Although the proposed non-hackability conditions ensure that higher correctness probabilities lead to higher optimal expected rewards, some strictly non-hackable reward schemes still result in substantial accuracy degradation after RL training. For instance, on DeepMath-103K with Qwen2.5-3B, Brier-log Hybrid achieves only 26.35% accuracy, compared with 42.38% for the correctness-only baseline. This suggests a considerable gap between the theoretical incentive guarantees and practical RL optimization outcomes. The authors should investigate the underlying causes of such degradation and clarify the extent to which the proposed theoretical framework can reliably guide reward function selection in practice.
Questions for authors
  1. Do the key comparisons and spectrum trends persist across independent training seeds and when invalid or truncated outputs are separated from valid-answer errors? A small representative subset would already be informative.
  2. How much of the spectrum's empirical ordering remains after accounting for reward variance or effective update size? Please distinguish the proven penalty asymmetry from the hypothesized effect of finite optimization.
  3. For HotpotQA-Modified examples with removed evidence, how do the authors distinguish reward-driven avoidable abstention from appropriate uncertainty under missing information? Please scope the claim to the evaluation convention that abstention counts as incorrect.
5 comments
AuthorsOpenReview ↗

Reviewer GPQ2 Rebuttal (Part 1)

W1: 'Insufficient evaluation of training stability and reproducibility'

Insufficient evaluation of training stability and reproducibility. The experimental results are primarily based on single training runs, without reporting means and standard deviations across multiple random seeds. Given the inherent instability of RL training, it is difficult to determine whether the observed differences between reward schemes are statistically reliable. Moreover, the authors acknowledge training instability and unsuccessful runs in Appendix I, which further raises concerns about the robustness of the reported results. I encourage the authors to conduct multi-seed experiments and report the mean performance, standard deviation, and training failure rate for each reward scheme.

Do the key comparisons and spectrum trends persist across independent training seeds and when invalid or truncated outputs are separated from valid-answer errors? A small representative subset would already be informative.

We thank you for your suggestion. We have repeated our experiments for Gemma 4 (E2B) Instruct three times. The exception is Brier-log Hybrid where the third attempt is on the process of evaluation. Results will be updated upon acceptance. The formatting reward is set to 0.5. To discourage overly long responses, an overlong penalty of 0 for response lengths at most 2048 tokens, 0.5 for response lengths at the maximum length of 4096 tokens, and linearly interpolated between 2048 and 4096 tokens, is set. We empirically found that this configuration is more stable, and less prone to training collapse without resorting to large formatting reward.

The criteria for training collapse are at least one of the following:

  • Training loss above 0.1 any time
  • Formatting error rate above 0.05 any time during training, or equivalently, a formatting reward below 0.95 of the maximum.
  • Reward decreases by at least 0.5 from peak. 0.5 is far beyond the typical reward variance during RL training. The criteria have been included in Appendix J of the main paper.

Of these 39 RL experiments, only 3 resulted in LLM training collapse. We have reported which reward schemes have collapsed in Appendix F for the main experiments and Appendix I for the ablation studies.

We have since updated Appendix F to include the new results. For ease of reference, the results of the repeated Gemma 4 (E2B) Instruct experiments are as follows, in the format of mean ± standard deviation:

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Base (JSON)0.3008 ± 0.00090.7453 ± 0.0009-0.1068 ± 0.00250.4076 ± 0.00170.4624 ± 0.0014-0.4624 ± 0.0014
Log Loss0.0000 ± 0.00000.0000 ± 0.0000-0.0002 ± 0.00010.0002 ± 0.00010.0083 ± 0.0057-0.0083 ± 0.0057
Brier Score0.0000 ± 0.00000.1603 ± 0.2777-0.0001 ± 0.00010.0001 ± 0.00010.0051 ± 0.0002-0.0051 ± 0.0002
Log-10.0116 ± 0.01010.7649 ± 0.25020.0055 ± 0.00670.0061 ± 0.00680.0069 ± 0.0017-0.0051 ± 0.0024
Brier-20.1336 ± 0.13670.8554 ± 0.13000.0763 ± 0.07250.0574 ± 0.06570.0533 ± 0.0573-0.0296 ± 0.0226
Log-1ln⁡202\frac{1}{\ln 202}0.4423 ± 0.00950.7509 ± 0.01180.2162 ± 0.01900.2261 ± 0.00970.1478 ± 0.0172-0.1208 ± 0.0626
Correctness-only0.4155 ± 0.01090.5618 ± 0.1059-0.1518 ± 0.04090.5673 ± 0.03020.5715 ± 0.0246-0.5715 ± 0.0246
Overconfidence-10000.4348 ± 0.01660.7386 ± 0.00960.1262 ± 0.09800.3086 ± 0.11240.2667 ± 0.2079-0.2641 ± 0.2120
Overconfidence-40.4405 ± 0.01130.7583 ± 0.01640.2095 ± 0.00140.2311 ± 0.01160.1580 ± 0.0253-0.1342 ± 0.0601
Overconfidence-10.4373 ± 0.00780.7734 ± 0.00540.2247 ± 0.00730.2126 ± 0.01460.1232 ± 0.0399-0.0590 ± 0.0398
Brier-10.4293 ± 0.00630.7742 ± 0.02050.2305 ± 0.00620.1988 ± 0.01170.0850 ± 0.0132-0.0220 ± 0.0367
Underconfidence-10.4390 ± 0.00380.7723 ± 0.01180.2361 ± 0.00380.2029 ± 0.00460.0876 ± 0.0258-0.0386 ± 0.0711
Underconfidence-40.4420 ± 0.01110.7799 ± 0.00600.2476 ± 0.01210.1944 ± 0.00470.0727 ± 0.0151-0.0423 ± 0.0238
Brier-log Hybrid0.4246 ± 0.02460.7591 ± 0.01380.2195 ± 0.02290.2051 ± 0.00580.0974 ± 0.0232-0.0687 ± 0.0695

(to be continued)

AuthorsOpenReview ↗

Reviewer GPQ2 Rebuttal (Part 2)

W2: Non-hackable confidence theoretically may not imply non-hackable confidence practically due to optimization dynamics.

Theoretical non-hackability does not necessarily translate into accuracy preservation in practice. Although the proposed non-hackability conditions ensure that higher correctness probabilities lead to higher optimal expected rewards, some strictly non-hackable reward schemes still result in substantial accuracy degradation after RL training. For instance, on DeepMath-103K with Qwen2.5-3B, Brier-log Hybrid achieves only 26.35% accuracy, compared with 42.38% for the correctness-only baseline. This suggests a considerable gap between the theoretical incentive guarantees and practical RL optimization outcomes. The authors should investigate the underlying causes of such degradation and clarify the extent to which the proposed theoretical framework can reliably guide reward function selection in practice.

The reason is likely due to the gradient conflict between accuracy and confidence calibration when they are jointly optimized, leading to a trade-off in accuracy and confidence calibration [1]. Each iteration, the LLM can only priortize accuracy, or confidence calibration, but not both. The overconfidence penalty is more severe relative to the underconfidence penalty gradient for non-hackable confidence reward schemes nearer to the underconfident end of the spectrum. This implies that the expected reward is likely higher prioritizing confidence calibration over accuracy nearer to the underconfident end of the spectrum. Therefore, reward schemes in the underconfident end of the spectrum prioritize confidence calibration, while reward schemes nearer to the overconfident end of the spectrum prioritize accuracy. We have since updated our paper to include this phenomenon as a limitation of our work, and included a more detailed and intuitive explanation of this phenomenon to Appendix H.2.


W3: Effect of effective batch size on overconfidence to underconfidence spectrum ordering

How much of the spectrum's empirical ordering remains after accounting for reward variance or effective update size? Please distinguish the proven penalty asymmetry from the hypothesized effect of finite optimization.

The trends in the overconfidence to underconfidence spectrum hold up to expected statistical noise of up to approximately 0.1, based on standard deviations mostly in the order of 0.01 to 0.1 when repeating Gemma 4 (E2B) Instruct three times. We have run experiments on Gemma 4 (E2B) Instruct using HotpotQA-Modified dataset. We have doubled the number of responses per question to 16 in one experiment, and doubled the number of questions per batch to 128 in a separate experiment. The results are as follows:

Doubled the number of responses per question to 16

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Log Loss0.0000--0.00010.00010.0050-0.0050
Brier Score0.0000-0.00000.00000.0050-0.0050
Log-10.0000-0.00000.00000.0050-0.0050
Brier-20.0007--0.00090.00160.0065-0.0065
Log-1ln⁡202\frac{1}{\ln 202}0.43220.75360.22540.20680.08850.0274
Correctness-only0.39800.5057-0.19440.59240.5939-0.5915
Overconfidence-10000.45720.7497-0.00240.45950.4846-0.4846
Overconfidence-40.44240.77340.19600.24640.1975-0.1933
Overconfidence-10.45220.79930.25510.19710.1132-0.0379
Brier-10.45070.76500.22720.22350.1647-0.0202
Underconfidence-10.45660.79740.27120.18540.0557-0.0125
Underconfidence-40.42490.75850.22690.19800.0516-0.0470
Brier-log Hybrid0.42780.77070.21560.21220.1308-0.0512

(to be continued)

AuthorsOpenReview ↗

Reviewer GPQ2 Rebuttal (Part 3)

W3 (continued)

Doubled the number of questions per batch to 128

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Log Loss0.0000--0.00080.00080.0057-0.0057
Brier Score0.0000-0.00000.00000.0050-0.0050
Log-10.01050.94370.00670.00380.0057-0.0056
Brier-20.31360.89420.19310.12050.0772-0.0626
Log-1ln⁡202\frac{1}{\ln 202}0.45660.74940.24060.21590.11460.0055
Correctness-only0.41410.5000-0.16600.58010.5809-0.5809
Overconfidence-10000.32790.5001-0.33750.66540.6671-0.6671
Overconfidence-40.45900.77740.19910.25990.2310-0.1796
Overconfidence-10.44400.77450.22220.22180.1362-0.1262
Brier-10.43370.75870.21980.21390.1345-0.0332
Underconfidence-10.44930.78130.25560.19380.0663-0.0196
Underconfidence-40.45670.77720.25500.20170.0952-0.0028
Brier-log Hybrid0.42320.74210.21740.20580.0834-0.0746

W4: The abstention claim should come with a caveat mentioning that abstentions are considered wrong answers.

For HotpotQA-Modified examples with removed evidence, how do the authors distinguish reward-driven avoidable abstention from appropriate uncertainty under missing information? Please scope the claim to the evaluation convention that abstention counts as incorrect.

Thank you for your suggestion. Assuming abstentions are considered wrong answers, all reward-driven abstention is avoidable due to the existence of non-hackable confidence reward schemes. We have updated the second paragraph of Section 3.1 to make it explicit that in our problem setting, abstentions are considered incorrect. In particular, abstention in our setting is avoidable due to the existence of non-hackable confidence reward schemes, therefore, the ideal rate of abstention is zero, and all abstention is reward-driven. However, due to various factors such as the gradient conflict between accuracy and confidence when they are jointly optimized [1], abstentions can exist even in non-hackable confidence reward schemes as LLMs prioritize confidence optimization to varying degrees. A more detailed explanation can be found in Appendix H.2.


References:

[1] Zhengzhao Ma, Xueru Wen, Boxi Cao, Yaojie Lu, Hongyu Lin, Jinglin Yang, Min He, Xianpei Han, and Le Sun. Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=3V1p2bJugq.

Reviewer_BHNwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reward function design for jointly optimizing reasoning accuracy and verbalized confidence calibration in LLMs. The authors identify confidence reward hacking, where poorly designed calibration rewards can incentivize models to intentionally produce incorrect answers. They introduce non-hackable confidence reward schemes and characterize them through interpretability, proper scoring, and best-effort conditions. The paper further connects these schemes to behavioral calibration and establishes an overconfidence–underconfidence spectrum based on reward penalties. Experiments across multiple LLMs and reasoning datasets demonstrate confidence reward hacking under certain reward schemes.

Critical strengths
  1. The paper provides a systematic framework for designing reward functions that simultaneously encourage answer correctness and calibrated confidence. In particular, Theorem 1 gives necessary and sufficient conditions for weakly non-hackable confidence reward schemes, extending the standard proper-scoring framework with a best-effort requirement. The connection to behavioral calibration in Theorem 2 further strengthens the theoretical contribution. These results provide useful guidance for reward design beyond specific choices such as Brier or logarithmic scoring.

  2. The paper presents empirical evidence that poorly designed confidence rewards can substantially reduce reasoning accuracy. For example, Table 2 shows severe accuracy degradation under Log-1 and Brier-2 rewards, particularly on challenging questions, while non-hackable reward schemes largely preserve accuracy. The observed calibration biases across different reward functions also broadly agree with the proposed theoretical spectrum.

Critical weaknesses
  1. The non-hackability guarantees do not directly characterize practical RL optimization. The proposed definition of non-hackability requires the optimal expected reward, after optimizing confidence, to increase with the probability of answering correctly. However, this assumes that the model can achieve optimal confidence for a given correctness probability. In practice, LLMs jointly learn reasoning and confidence through gradient-based RL, and their confidence estimates may remain substantially miscalibrated throughout training. Consequently, a reward scheme satisfying the proposed non-hackability conditions may not necessarily encourage accuracy improvements under imperfect confidence predictions or during intermediate optimization stages. The theoretical guarantees therefore do not directly establish robustness against undesirable behavior in practical RL training.

  2. The empirical comparison does not sufficiently disentangle reward incentives from optimization effects. Different reward schemes have substantially different reward scales, gradients, and variances, which can influence RL training dynamics independently of their theoretical non-hackability properties. Although the authors normalize some reward endpoints, this does not necessarily ensure comparable optimization behavior. For instance, the substantial accuracy degradation observed under Log-1 and Brier-2 is consistent with confidence reward hacking, but additional evidence would help distinguish deliberate reward-maximizing behavior from training instability or unfavorable reward scaling.

Questions for authors
  1. The non-hackability conditions are derived assuming optimal confidence for each correctness probability. Can the authors clarify whether these guarantees extend to settings where the model's confidence is imperfectly calibrated?

  2. How do the authors distinguish confidence reward hacking from optimization instability or differences in reward scales and variances? Could the authors provide additional analyses of reward distributions, advantage statistics, or training dynamics for hackable and non-hackable reward schemes?

  3. How sensitive are the overconfidence–underconfidence spectrum results to RL hyperparameters and training duration?

  4. The paper demonstrates that certain reward schemes preserve accuracy while improving calibration. Is there a principled criterion for selecting among different non-hackable reward schemes in practice, particularly when balancing accuracy, calibration error, and discrimination?

6 comments
AuthorsOpenReview ↗

Reviewer BHNw Rebuttal (Part 1)

W1: Non-hackable confidence reward schemes may not guarantee best effort due to assumption of perfect confidence calibration.

The non-hackability guarantees do not directly characterize practical RL optimization. The proposed definition of non-hackability requires the optimal expected reward, after optimizing confidence, to increase with the probability of answering correctly. However, this assumes that the model can achieve optimal confidence for a given correctness probability. In practice, LLMs jointly learn reasoning and confidence through gradient-based RL, and their confidence estimates may remain substantially miscalibrated throughout training. Consequently, a reward scheme satisfying the proposed non-hackability conditions may not necessarily encourage accuracy improvements under imperfect confidence predictions or during intermediate optimization stages. The theoretical guarantees therefore do not directly establish robustness against undesirable behavior in practical RL training.

The non-hackability conditions are derived assuming optimal confidence for each correctness probability. Can the authors clarify whether these guarantees extend to settings where the model's confidence is imperfectly calibrated?

We appreciate your thoughtful question. Theoretically, non-hackable confidence reward schemes always prefer correct answers to wrong answers regardless of confidence. By Theorem 1(iii), all weakly non-hackable confidence reward schemes have reward for correctness f(c)f(c) at least the reward for incorrectness g(c)g(c) for any fixed confidence c. In particular, strictly non-hackable confidence reward schemes satisfy strict interpretability for all confidence cc in (0,1)(0, 1). Therefore, f(0+)≥g(0+)f(0^+) \geq g(0^+) implies f(c)>g(d)f(c) > g(d) for all confidences c,dc, d in (0,1)(0, 1). Therefore, we can guarantee for strict non-hackable confidence reward schemes, even if the LLM is severely miscalibrated in confidence, the LLM is incentivized to answer correctly. We have since updated the paper to include this theoretical guarantee as Theorem 2, with proof in Appendix A3.

The gap between theoretical incentives and practical RL optimization is included as a limitation. Our theory of non-hackable confidence reward schemes only concerns the design of reward functions or utility functions, and not about the optimization dynamics of RL. The RL experiments in our current paper are intended as a proof of concept to show the presence of confidence reward hacking and the various calibration biases that could result by training using non-hackable confidence reward schemes. A more detailed exploration on the effects of optimization for RL on different non-hackable confidence reward schemes is left to future work. We have since updated our paper to clarify this point.


W2: Empirical evidence is not fully consistent with confidence reward hacking due to RL training optimization dynamics

The empirical comparison does not sufficiently disentangle reward incentives from optimization effects. Different reward schemes have substantially different reward scales, gradients, and variances, which can influence RL training dynamics independently of their theoretical non-hackability properties. Although the authors normalize some reward endpoints, this does not necessarily ensure comparable optimization behavior. For instance, the substantial accuracy degradation observed under Log-1 and Brier-2 is consistent with confidence reward hacking, but additional evidence would help distinguish deliberate reward-maximizing behavior from training instability or unfavorable reward scaling.

We appreciate your suggestion. The reward, excluding formatting and overlong rewards, increases with decreases in accuracy in the event of confidence reward hacking, while the reward increases with increases in accuracy when confidence reward hacking does not happen. If the reward decreases significantly beyond the typical reward variance, this will be considered training collapse. We have added training statistics regarding the reward (excluding formatting-related rewards) and accuracy to Section 5.1 to clarify the point, as well as an explanation in the third-last paragraph in the section.

We will release the Weights and Biases (WandB) training statistics upon paper acceptance. The training statistics contain the training losses, each component of the reward, accuracy statistics and output length statistics for every step of the training process.


(to be continued)

AuthorsOpenReview ↗

Reviewer BHNw Rebuttal (Part 2)

W3: Unclear evidence to differentiate between confidence reward hacking, optimization instability, and reward scale differences.

How do the authors distinguish confidence reward hacking from optimization instability or differences in reward scales and variances? Could the authors provide additional analyses of reward distributions, advantage statistics, or training dynamics for hackable and non-hackable reward schemes?

As for different reward scales, we have tested different formatting reward values of 0.5 and 5 for Qwen 3 (4B) DeepMath-103K while keeping the reward scheme the same, and found the degrees of confidence reward hacking largely similar in hackable confidence reward schemes. With a higher formatting reward of 5, the accuracy was slightly lower, but the AUROC was generally slightly better. The overconfidence to underconfidence spectrum trends generally held after allowing for an assumed statistical noise of up to 0.1 in the calibration bias. The estimate of 0.1 is obtained from estimates of the standard deviations of the calibration bias from the three runs of Gemma 4 (E2B) Instruct HotpotQA-Modified.

The results are as follows:

Formatting reward 0.5

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Log Loss0.0000-0.00000.00000.0050-0.0050
Brier Score0.00230.92060.00110.00120.0054-0.0052
Log-10.80190.74400.66600.13590.0226-0.0140
Brier-20.79040.76060.65210.13840.03080.0058
Log-1ln⁡202\frac{1}{\ln 202}0.81540.71880.68160.13380.0275-0.0133
Correctness-only0.82600.73620.67790.14820.1319-0.1311
Overconfidence-10000.80460.70390.66220.14240.0222-0.0174
Overconfidence-40.81380.70270.67720.13660.03370.0128
Overconfidence-10.83580.67960.70540.13040.04840.0436
Brier-10.81170.75380.68150.13020.04640.0379
Underconfidence-10.78790.74020.64330.14460.03320.0052
Underconfidence-40.78870.74160.64540.14330.0201-0.0117
Brier-log Hybrid0.75760.74040.59660.16100.0447-0.0428

Formatting reward 5

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Log Loss0.0000--0.00010.00010.0051-0.0051
Brier Score0.00350.85960.00050.00300.0050-0.0049
Log-10.71760.74010.53960.17800.0763-0.0762
Brier-20.75340.76330.60170.15170.0305-0.0106
Log-1ln⁡202\frac{1}{\ln 202}0.73400.73690.56450.16950.0369-0.0288
Correctness-only0.73270.75590.55660.17610.1186-0.1186
Overconfidence-10000.74990.76150.59140.15850.0591-0.0591
Overconfidence-40.77290.76200.62850.14450.0316-0.0210
Overconfidence-10.74800.74270.58530.16260.0397-0.0363
Brier-10.77830.73180.62650.15190.05610.0411
Underconfidence-10.74810.76480.59390.15420.0325-0.0288
Underconfidence-40.75720.75440.60240.15470.0324-0.0226
Brier-log Hybrid0.71950.73140.54240.17710.0546-0.0546

(to be continued)

AuthorsOpenReview ↗

Reviewer BHNw Rebuttal (Part 3)

W4: Lack of ablation sensitivity analysis to RL hyperparameters, especially training duration

How sensitive are the overconfidence–underconfidence spectrum results to RL hyperparameters and training duration?

We have run experiments on Gemma 4 (E2B) Instruct using HotpotQA-Modified dataset. We changed the following:

  • A: Doubled the number of responses per question to 16
  • B: Doubled the number of questions per batch to 128
  • C: Evaluated the responses at 20 steps and 40 steps during the second run, during which there were no training collapses.

The results are as follows:

A: Doubled the number of responses per question to 16

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Log Loss0.0000--0.00010.00010.0050-0.0050
Brier Score0.0000-0.00000.00000.0050-0.0050
Log-10.0000-0.00000.00000.0050-0.0050
Brier-20.0007--0.00090.00160.0065-0.0065
Log-1ln⁡202\frac{1}{\ln 202}0.43220.75360.22540.20680.08850.0274
Correctness-only0.39800.5057-0.19440.59240.5939-0.5915
Overconfidence-10000.45720.7497-0.00240.45950.4846-0.4846
Overconfidence-40.44240.77340.19600.24640.1975-0.1933
Overconfidence-10.45220.79930.25510.19710.1132-0.0379
Brier-10.45070.76500.22720.22350.1647-0.0202
Underconfidence-10.45660.79740.27120.18540.0557-0.0125
Underconfidence-40.42490.75850.22690.19800.0516-0.0470
Brier-log Hybrid0.42780.77070.21560.21220.1308-0.0512

B: Doubled the number of questions per batch to 128

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Log Loss0.0000--0.00080.00080.0057-0.0057
Brier Score0.0000-0.00000.00000.0050-0.0050
Log-10.01050.94370.00670.00380.0057-0.0056
Brier-20.31360.89420.19310.12050.0772-0.0626
Log-1ln⁡202\frac{1}{\ln 202}0.45660.74940.24060.21590.11460.0055
Correctness-only0.41410.5000-0.16600.58010.5809-0.5809
Overconfidence-10000.32790.5001-0.33750.66540.6671-0.6671
Overconfidence-40.45900.77740.19910.25990.2310-0.1796
Overconfidence-10.44400.77450.22220.22180.1362-0.1262
Brier-10.43370.75870.21980.21390.1345-0.0332
Underconfidence-10.44930.78130.25560.19380.0663-0.0196
Underconfidence-40.45670.77720.25500.20170.0952-0.0028
Brier-log Hybrid0.42320.74210.21740.20580.0834-0.0746

(to be continued)

Reviewer_uJN4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reward design for jointly training large language models (LLMs) to improve answer accuracy and verbalized confidence calibration through reinforcement learning. It identifies confidence reward hacking, where poorly designed confidence-dependent rewards incentivize models to answer incorrectly or abstain in order to obtain higher rewards. The authors formalize non-hackable confidence reward schemes using interpretability, proper-scoring, and best-effort properties, and provide a characterization of weakly non-hackable schemes. They further connect this framework to behavioral calibration and construct a spectrum of reward schemes that exhibit different overconfidence–underconfidence tendencies. Experiments on HotpotQA, a modified HotpotQA dataset, BigMath, and DeepMath-103K, using Qwen and Gemma models, provide empirical evidence of reward hacking under certain reward schemes and show that different non-hackable schemes induce different calibration biases.

Critical strengths
  • The paper investigates confidence reward hacking in RL-based confidence calibration, which is an important issue for developing more reliable LLMs. Poorly designed reward functions may incentivize undesirable behaviors that undermine the intended objectives of improving reasoning accuracy and confidence calibration. Explicitly characterizing these failure modes is valuable for the design of confidence-aware RL algorithms.
  • The paper formalizes non-hackable confidence reward schemes through properties including interpretability, proper scoring, and best effort. It also characterizes weakly non-hackable schemes and analyzes how reward design relates to overconfidence and underconfidence. This provides a useful theoretical perspective for understanding the incentives induced by different confidence rewards, rather than relying solely on empirical trial and error.
  • The experiments investigate reward functions with different tendencies toward overconfidence and underconfidence across multiple datasets and model configurations. The results suggest that reward design can systematically influence calibration bias and that improved calibration may come with a reduction in accuracy or an increase in abstention, depending on the reward scheme and task. This comparison provides useful insights into the trade-offs involved in confidence-aware RL.
Critical weaknesses
  • The paper does not sufficiently analyze alternative approaches that decouple reasoning optimization from confidence calibration, such as DCPO (Ma et al., 2026). Although §2.1 discusses its calibration reward and lack of proper scoring, it does not address the central mechanism of assigning separate optimization signals to answer and confidence tokens. Such decoupling may mitigate the identified failure mode by preventing calibration rewards from directly reinforcing incorrect answers. Consequently, the analysis of joint scalar rewards alone does not establish whether the same failure persists under decoupled optimization.
  • The paper's definition of non-hackability provides a principled way to characterize the incentives induced by a reward function under the assumed probabilistic setting. However, desirable properties of the optimal expected-reward strategy do not guarantee that finite-step RL optimization will follow a trajectory that preserves or improves answer accuracy. In practice, the optimization process is affected by the initial policy, finite training budgets, sampling variability, and the scale and direction of policy updates. Consequently, a theoretically non-hackable reward scheme may still exhibit undesirable intermediate behaviors or fail to achieve the intended calibration outcome. *The empirical evidence is not presented consistently across models and datasets. In particular, the main paper places limited emphasis on quantitative calibration metrics such as ECE and Brier score, while more detailed results are deferred to the appendix and receive relatively little discussion. The calibration-bias plots also provide an incomplete basis for comparing model behavior across datasets and model families.
  • The paper attributes certain accuracy degradations and increases in abstention to confidence reward hacking. While these observations are consistent with the proposed explanation, changes in accuracy alone do not uniquely identify reward hacking. Different reward functions can induce different optimization signals and policy-update trajectories, and their effects may also depend on the model, dataset, and training dynamics.
Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Reviewer uJN4 Rebuttal (Part 1)

W1. Lack of consideration of DCPO [1], which considers accuracy and confidence calibration trained separately, and may mitigate confidence reward hacking.

The paper does not sufficiently analyze alternative approaches that decouple reasoning optimization from confidence calibration, such as DCPO (Ma et al., 2026). Although §2.1 discusses its calibration reward and lack of proper scoring, it does not address the central mechanism of assigning separate optimization signals to answer and confidence tokens. Such decoupling may mitigate the identified failure mode by preventing calibration rewards from directly reinforcing incorrect answers. Consequently, the analysis of joint scalar rewards alone does not establish whether the same failure persists under decoupled optimization.

Response to W1:

We appreciate your feedback and will incorporate your suggestion into our paper whenever possible.

DCPO may not completely mitigate confidence reward hacking

In DCPO, the paradigm of decoupling reasoning tokens from confidence tokens in RL training refers to the separation of accuracy and confidence calibration tokens for the respective accuracy and confidence calibration rewards [1]. Accuracy and confidence calibration are rewarded separately, and losses are only taken only over the reasoning and confidence calibration tokens respectively. The accuracy and confidence calibration loss terms are weighted by their respective advantages. The loss terms are then summed to obtain the final GRPO loss [2].

The use of standard deviation normalized advantages in DCPO helps ensure a consistent reward scale where one loss term does not consistently dominate the other. However, standard deviation normalized advantages have been demonstrated by [3] to incentivize misreporting of LLM confidence. Therefore, another form of reward hacking may occur, when the LLM is incentivized to misreport its confidence. However, we did not have the time to study this experimentally during the rebuttal period.

DCPO may be effective with non-hackable confidence reward scheme as calibration reward

On the other hand, decoupled optimization may be more effective in solving the optimization problem because the accuracy loss terms are not affected by the LLM confidence tokens, which mitigates the entangling when both accuracy and confidence tokens are optimized over a single joint reward scheme [1]. If this is the case, then using decoupled optimization with a non-hackable confidence reward scheme as the calibration reward and with standard deviation normalization disabled could be highly effective.

DCPO viewed as credit assignment

In response to this reviewer's query, we frame the decoupling mechanism in DCPO as a form of credit assignment to separate accuracy and confidence calibration rewards. Subsequently, we demonstrate the possibility in Appendix A.8 that any non-hackable confidence reward scheme can be decomposed into the sum of an accuracy reward component and a proper scoring calibration reward. Therefore, for any non-hackable confidence reward scheme, there exists a credit assignment to accuracy and confidence calibration tokens such that the LLM is incentivized to truthfully report confidence and to maximize its accuracy.

It is also possible to combine both an accuracy reward and a proper scoring calibration reward into a non-hackable confidence reward scheme by summing both components. If the calibration reward satisfies proper scoring, [4] provides a lower bound for the accuracy reward in the case of a correct answer to satisfy best effort. Since the addition of an accuracy reward does not affect proper scoring [4] and proper scoring implies interpretability [5], best effort is satisfied implies the resulting reward scheme is non-hackable confidence.

We have since included an entire subsection in Appendix A.8 regarding the applicability of our work to DCPO.

Inclusion in 'Limitations and Future Work'

However, an empirical study on whether the decoupled form or the joint form is better is beyond the intended scope of our work because the intended main contribution is about the theoretical incentive of reward or utility functions which jointly reward accuracy and confidence calibration. We have since included this as the fifth limitation in the 'Limitations and Future Work' section.


(to be continued)

AuthorsOpenReview ↗

Reviewer uJN4 Rebuttal (Part 2)

W2. Non-hackable confidence reward schemes do not guarantee no confidence reward hacking in practice

The paper's definition of non-hackability provides a principled way to characterize the incentives induced by a reward function under the assumed probabilistic setting. However, desirable properties of the optimal expected-reward strategy do not guarantee that finite-step RL optimization will follow a trajectory that preserves or improves answer accuracy. In practice, the optimization process is affected by the initial policy, finite training budgets, sampling variability, and the scale and direction of policy updates. Consequently, a theoretically non-hackable reward scheme may still exhibit undesirable intermediate behaviors or fail to achieve the intended calibration outcome.

We appreciate your feedback of our paper. The accuracy to confidence calibration tradeoffs observed in Appendix H.2 is one such example where theoretically non-hackable confidence reward schemes may not lead to optimal training accuracy, especially for reward schemes near the underconfident end of the spectrum. We have since included an explanation of the phenomenon in Appendix H.2, i.e. the gradient conflict between accuracy and confidence calibration objectives, which slows down training for accuarcy and/or confidence calibration [1].

However, our paper is intended to be primarily about the theoretical design of reward or utility functions that provide both accuracy and confidence calibration incentives, solving the failure mode where a poorly designed reward function incentivizes sacrificing accuracy for better confidence calibration. The RL experiments are intended as a proof of concept to demonstrate the extent in which theoretical incentives hold in practice. The failure of empirical RL optimization to maximize the expected reward is a separate failure mode and is beyond the scope of the paper. We have since included this as a limitation of the paper in the 'Limitations and Further Work' section.


W3. Inconsistent presentation of empirical evidence

The empirical evidence is not presented consistently across models and datasets. In particular, the main paper places limited emphasis on quantitative calibration metrics such as ECE and Brier score, while more detailed results are deferred to the appendix and receive relatively little discussion. The calibration-bias plots also provide an incomplete basis for comparing model behavior across datasets and model families.

We appreciate your feedback and have indeed found an experimental inconsistency. The evidence in Qwen 3 (4B) DeepMath-103K appeared to be wildly inconsistent with the overconfidence to underconfidence spectrum. This was due to a mistake where we used a format enforcer to grade the LLM outputs in evaluation but not during RL training, which negatively affected the invalid formatting rates. We have since removed the format enforcer and reran the evaluation. The new results are more consistent with the overconfidence to underconfidence spectrum trends as described in the paper. We have updated our results in Appendix F.

To account for statistical variance between runs, we added error bars in Figure 4 for Gemma 4 (E2B) Instruct HotpotQA-Modified, which we ran three times. The results are consistent with the overconfidence to underconfidence spectrum after accounting for the statistical variance. For Gemma 4 (E2B) Instruct HotpotQA-Modified, there is considerable variability in the calibration bias, with standard deviations usually in the order of 0.01 to 0.1 and sometimes higher. Therefore, single-run variations up to approximately 0.1 in calibration bias, either direction, are to be expected.

As the overconfidence to underconfidence spectrum is designed to get the LLM to bias either towards higher or lower confidence whenever unsure due to optimizer limitations, we find the ECE and Brier score inappropriate. A high ECE or a high Brier score cannot be used to differentiate if the LLM is consistently overconfident or the LLM is consistently underconfident. Nonetheless, we are pleased that you have found the ECE and Brier score statistics in Appendix F and Appendix H useful. We have since added some explanation of the results using the theory in DCPO [1] in Appendix H.2.


(to be continued)

AuthorsOpenReview ↗

Reviewer uJN4 Rebuttal (Part 3)

W4. Abstention rate increase and lower test-time accuracies do not necessarily imply confidence reward hacking

The paper attributes certain accuracy degradations and increases in abstention to confidence reward hacking. While these observations are consistent with the proposed explanation, changes in accuracy alone do not uniquely identify reward hacking. Different reward functions can induce different optimization signals and policy-update trajectories, and their effects may also depend on the model, dataset, and training dynamics.

We appreciate your suggestion, and you are spot on. We have since included training dynamics statistics in Section 5.1 showing the increasing reward and the decreasing accuracy during training for hackable confidence reward schemes, suggesting that the LLM is sacrificing accuracy for better overall reward due to better overall confidence calibration, i.e. confidence reward hacking.

Upon paper acceptance, we will also release detailed per-step training statistics by making our Weights and Biases (WandB) results public. The training statistics contain the training losses, each component of the reward, accuracy statistics and output length statistics for every step of the training process.


References:

[1] Zhengzhao Ma, Xueru Wen, Boxi Cao, Yaojie Lu, Hongyu Lin, Jinglin Yang, Min He, Xianpei Han, and Le Sun. Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=3V1p2bJugq.

[2] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024.

[3] Michael Bereket and Jure Leskovec. Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes. arXiv preprint arXiv:2508.11800, 2025.

[4] Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=ASQ649zdHm.

[5] Mark J Schervish. A General Method for Comparing Probability Assessors. The Annals of Statistics, 17(4):1856–1879, 1989.

Thank you for your time and thoughtful consideration. As the deadline approaches, we would be grateful for any feedback you may have on our response. We would be happy to clarify any points or answer any questions.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Global Comment (Part 1)

We thank all the reviewers for their helpful comments. We would like to reiterate the focus of the paper and we hope that the reviewers would consider the paper from the view of its intended contributions.

Key contributions

1. Definition and Characterization of Non-hackable Confidence: We define non-hackable confidence reward schemes as reward schemes that reward more for confidently correct answers than unconfidently correct answers, penalize more for confidently incorrect answers than unconfidently incorrect answers, encourage the LLM to be truthful about its confidence and to answer to the best of its ability. We generalize the approach of Damani et. al. [1] from a sum of correctness and calibration rewards to two confidence dependent reward functions, respectively for correct and incorrect answers. We derive necessary and sufficient conditions under which they are weakly non-hackable confidence, and sufficient conditions for them to be strongly non-hackable confidence.

2. Behavioral Calibration is a Special Case of Non-hackable Confidence[2,3,4]: We unify the non-hackable confidence approach with the approach based on behavioral calibration with abstention initiated by Kalai et. al. [2], which provides 0 reward for abstaining, a fixed positive reward for answering correctly, and an abstention-threshold dependent penalty for answering incorrectly. We do that by proving that the constructions based on the behavioral calibration approach from [3, 4] result in non-hackable confidence reward schemes.

3. Design of overconfidence to underconfidence spectrum: We introduce an overconfidence to underconfidence spectrum of reward schemes. We give sufficient conditions for a reward scheme to have overconfidence or underconfidence bias. We design two families of reward schemes, each containing a single parameter, namely, Overconfidence-k and Underconfidence-k. Overconfidence-k has overconfidence bias, with larger k meaning larger degree of overconfidence biase. Underconfidence-k has underconfidence bias, with larger k meaning larger degree of underconfidence bias.

Clarification of Scope

1. Theory of reward functions extend beyond RL: The three key contributions are on the theory on reward functions. As long as there is a reward or utility function, the theory can be applied. Apart from reinforcement learning (RL), it can be applied to evolutionary settings [5, 6] and LLM evaluation [4].

2. Main focus is understanding confidence reward hacking as a failure mode: Reward functions only incentivize the behavior of the agents trained on them and do not guarantee correct behavior as approximation, optimization, and generalization properties are also important in the training process. However, well-designed reward functions eliminate a possible failure mode, i.e. confidence reward hacking, hence it is important to understand them.

3. Empirical experiments are proof-of-concept: We provide experimental results that corroborate the theory, although it is not exhaustive and does not definitively isolate the effects of reward function from approximation, optimization, and generalization properties of the training process. In particular, our experiment results are consistent with the possibility of confidence reward hacking occurring in practical datasets for hackable reward schemes and with different schemes inducing the corresponding calibration biases on the datasets. The same experiments provide indications on other properties of the different reward schemes, such as ECE [7] — we include these results in the appendix instead of leaving them out, although they do not directly corroborate the theoretical results.

(to be continued)

AuthorsOpenReview ↗

Global Comment (Part 2)

Experimental results

To respond to reviewer concerns, we have conducted the following experiments:

  1. Qwen 3 (4B) DeepMath-103K, formatting reward 0.5 (redone due to unsuitable setting in evaluation protocol causing high invalid formatting rates)
  2. Qwen 3 (4B) DeepMath-103K, formatting reward 5 (different reward scale)
  3. Gemma 4 (E2B) Instruct HotpotQA-Modified (repeated 3 times)
  4. Gemma 4 (E2B) Instruct HotpotQA-Modified at 20 and 40 steps instead of 60
  5. Gemma 4 (E2B) Instruct HotpotQA-Modified with responses per question doubled to 16
  6. Gemma 4 (E2B) Instruct HotpotQA-Modified with questions per batch doubled to 128

Sometimes, for Gemma 4 (E2B) Instruct experiments, LLM training collapse may occur. We use the following criteria for training collapse. If at least one of the following criteria are met, the run is considered anomalous and the experiment rerun.

  • Training loss exceeds 0.1 at any point during training
  • Formatting accuracy decreases below 0.95 at any point during training
  • Total reward decreases by more than 0.5 from peak, which is far greater than typical reward variance

Under all our scenarios, the trends in the overconfidence to underconfidence spectrum of reward schemes held up to statistical noise. The expected single-run statistical noise is expected to be up to approximately 0.1 in calibration bias because the observed standard deviations in our runs of Gemma 4 (E2B) Instruct are mostly in the order of 0.01 to 0.1 for each reward scheme.

1. Qwen 3 (4B) DeepMath-103K, formatting reward 0.5

The high invalid formatting rates were due to our evaluation setting that enabled the format enforcer. We disabled the format enforcer and all invalid formatting rates were reduced to below 0.1. The results are as follows:

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Base (JSON)0.43660.70950.01820.41850.4435-0.4435
Log Loss0.0000-0.00000.00000.0050-0.0050
Brier Score0.00230.92060.00110.00120.0054-0.0052
Log-10.80190.74400.66600.13590.0226-0.0140
Brier-20.79040.76060.65210.13840.03080.0058
Log-1ln⁡202\frac{1}{\ln 202}0.81540.71880.68160.13380.0275-0.0133
Correctness-only0.82600.73620.67790.14820.1319-0.1311
Overconfidence-10000.80460.70390.66220.14240.0222-0.0174
Overconfidence-40.81380.70270.67720.13660.03370.0128
Overconfidence-10.83580.67960.70540.13040.04840.0436
Brier-10.81170.75380.68150.13020.04640.0379
Underconfidence-10.78790.74020.64330.14460.03320.0052
Underconfidence-40.78870.74160.64540.14330.0201-0.0117
Brier-log Hybrid0.75760.74040.59660.16100.0447-0.0428

2. Qwen 3 (4B) DeepMath-103K, formatting reward 5

Like in Experiment 1, we disabled the format enforcer. The calibration bias trends in the overconfidence to underconfidence spectrum are largely similar up to expected statistical noise. The results are as follows:

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Log Loss0.0000--0.00010.00010.0051-0.0051
Brier Score0.00350.85960.00050.00300.0050-0.0049
Log-10.71760.74010.53960.17800.0763-0.0762
Brier-20.75340.76330.60170.15170.0305-0.0106
Log-1ln⁡202\frac{1}{\ln 202}0.73400.73690.56450.16950.0369-0.0288
Correctness-only0.73270.75590.55660.17610.1186-0.1186
Overconfidence-10000.74990.76150.59140.15850.0591-0.0591
Overconfidence-40.77290.76200.62850.14450.0316-0.0210
Overconfidence-10.74800.74270.58530.16260.0397-0.0363
Brier-10.77830.73180.62650.15190.05610.0411
Underconfidence-10.74810.76480.59390.15420.0325-0.0288
Underconfidence-40.75720.75440.60240.15470.0324-0.0226
Brier-log Hybrid0.71950.73140.54240.17710.0546-0.0546

(to be continued)

AuthorsOpenReview ↗

Global Comment (Part 3)

3. Gemma 4 (E2B) Instruct HotpotQA-Modified (repeated 3 times)

We empirically found this RL training configuration more stable without resorting to a larger formatting reward. 3 out of 39 of our training runs resulted in RL training collapse in the first attempt, namely Correctness-only in the first run, Overconfidence-4 in the third run and Brier-log Hybrid in the third run.

Regarding RL training collapse, RL training for Correctness-only in the first run succeeded on the third attempt. RL training for Overconfidence-4 in the third run and Brier-log Hybrid in the third run succeeded on the second attempt. All experiments in Gemma 4 (E2B) Instruct HotpotQA-Modified have been repeated three times. The results are as follows, with values expressed in the format mean ± standard deviation:

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Base (JSON)0.3008 ± 0.00090.7453 ± 0.0009-0.1068 ± 0.00250.4076 ± 0.00170.4624 ± 0.0014-0.4624 ± 0.0014
Log Loss0.0000 ± 0.00000.0000 ± 0.0000-0.0002 ± 0.00010.0002 ± 0.00010.0083 ± 0.0057-0.0083 ± 0.0057
Brier Score0.0000 ± 0.00000.1603 ± 0.2777-0.0001 ± 0.00010.0001 ± 0.00010.0051 ± 0.0002-0.0051 ± 0.0002
Log-10.0116 ± 0.01010.7649 ± 0.25020.0055 ± 0.00670.0061 ± 0.00680.0069 ± 0.0017-0.0051 ± 0.0024
Brier-20.1336 ± 0.13670.8554 ± 0.13000.0763 ± 0.07250.0574 ± 0.06570.0533 ± 0.0573-0.0296 ± 0.0226
Log-1ln⁡202\frac{1}{\ln 202}0.4423 ± 0.00950.7509 ± 0.01180.2162 ± 0.01900.2261 ± 0.00970.1478 ± 0.0172-0.1208 ± 0.0626
Correctness-only0.4155 ± 0.01090.5618 ± 0.1059-0.1518 ± 0.04090.5673 ± 0.03020.5715 ± 0.0246-0.5715 ± 0.0246
Overconfidence-10000.4348 ± 0.01660.7386 ± 0.00960.1262 ± 0.09800.3086 ± 0.11240.2667 ± 0.2079-0.2641 ± 0.2120
Overconfidence-40.4405 ± 0.01130.7583 ± 0.01640.2095 ± 0.00140.2311 ± 0.01160.1580 ± 0.0253-0.1342 ± 0.0601
Overconfidence-10.4373 ± 0.00780.7734 ± 0.00540.2247 ± 0.00730.2126 ± 0.01460.1232 ± 0.0399-0.0590 ± 0.0398
Brier-10.4293 ± 0.00630.7742 ± 0.02050.2305 ± 0.00620.1988 ± 0.01170.0850 ± 0.0132-0.0220 ± 0.0367
Underconfidence-10.4390 ± 0.00380.7723 ± 0.01180.2361 ± 0.00380.2029 ± 0.00460.0876 ± 0.0258-0.0386 ± 0.0711
Underconfidence-40.4420 ± 0.01110.7799 ± 0.00600.2476 ± 0.01210.1944 ± 0.00470.0727 ± 0.0151-0.0423 ± 0.0238
Brier-log Hybrid0.4246 ± 0.02460.7591 ± 0.01380.2195 ± 0.02290.2051 ± 0.00580.0974 ± 0.0232-0.0687 ± 0.0695

Update from last edit: Experiment completed with results tabulated

4. Gemma 4 (E2B) Instruct HotpotQA-Modified at 20 and 40 steps instead of 60

We took the second run of Experiment 3 and evaluated at steps 20 and 40 to simulate shorter training duration. The second run had no training collapse.

The results at step 20 are as follows:

Model/Reward SchemeAccuracyAUROCBrier-1BrierECECalib. Bias
Log Loss0.0000--0.00020.00020.0051-0.0051
Brier Score0.0000--0.00020.00020.0051-0.0051
Log-10.01680.79070.00300.01370.0081-0.0030
Brier-20.00000.6999-0.00010.00020.0050-0.0050
Log-1ln⁡202\frac{1}{\ln 202}0.44650.75680.22100.22550.1521-0.1501
Correctness-only0.41270.5014-0.16860.58140.5823-0.5823
Overconfidence-10000.44860.73520.01740.43120.4650-0.4650
Overconfidence-40.44890.73910.20680.24200.1696-0.1630
Overconfidence-10.43760.77490.21700.22060.1415-0.1026
Brier-10.43070.76450.22450.20620.0931-0.0411
Underconfidence-10.43720.76450.23140.20580.09730.0401
Underconfidence-40.43770.77610.23860.19910.0876-0.0188
Brier-log Hybrid0.43370.75640.22120.21260.1224-0.1219

(to be continued)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.