acceptodds
Withdrawn from ICLR 2027

Less Data Approximates More: Earning Faithful Confidence in High-Stakes Domains

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models are increasingly deployed in high-stakes domains, where confident yet incorrect inferences may cause severe real-world harm, bringing the long-overlooked issue of confidence faithfulness to the forefront. A promising solution jointly optimizes unsupervised Reinforcement Learning from Internal Feedback (RLIF) with reasoning-trace-guided Reasoning Distillation (RD), yet it faces three persistent challenges, namely the scarcity of high-quality training corpora, factually unwarranted overconfidence, and erroneous updates amplified by indiscriminate fusion. Inspired by how human confidence accumulates from uncertainty to certainty, we propose Progressive Reasoning Gain (PRG) to measure whether reasoning steps progressively strengthen confidence in the final answer. Building on PRG, we introduce HyTuning, a hybrid post-training framework that adaptively reweights RD and RLIF, using scarce supervised reasoning traces as a stable anchor while exploiting abundant unlabeled queries for scalability. Experiments on several domain-specific and general benchmarks demonstrate that HyTuning improves accuracy while achieving confidence faithfulness under limited supervision, supporting a practical “Less Data Approximates More” effect. Our code is available at: https://anonymous.4open.science/r/HyTuning-Code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training, Faithful Confidence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YeeKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HyTuning, a hybrid post-training framework combining reasoning distillation with reinforcement learning from internal feedback under limited supervision. Progressive Reasoning Gain (PRG) measures how confidence in the generated answer develops along a reasoning trace and adjusts the two objectives on queries with stored targets. Experiments cover cybersecurity and financial reasoning, model scaling, component ablations, and a blinded expert evaluation of PRG, while selective-prediction metrics assess the reliability of high-confidence answers.

Critical strengths
  1. The paper addresses the practically relevant problem of unreliable high-confidence answers under limited supervision.
  2. HyTuning shows empirical gains across cybersecurity and financial reasoning tasks and multiple model families.
  3. A blinded expert study provides useful evidence that low PRG is associated with weakly supported reasoning.
Critical weaknesses
  1. The main comparisons are not sufficiently controlled or comprehensive. Under partial supervision in Table 1, HyTuning uses both 100 verified traces and 3,000 synthetic queries, whereas each baseline accesses only one of these sources; OPD and OPSD also use different teacher resources (Appendix B.6). These differences make the rankings difficult to interpret as fair comparisons of training methods. Although Table 8 includes an entropy-based RLIF variant, a majority-vote reward baseline adapted from TTRL to the same training queries would provide a more complete evaluation.
  2. The main conclusions rely heavily on Qwen2.5-3B. Table 3 reports only accuracy improvements over the original backbones. Comparing HyTuning with the strongest partial-supervision baseline on additional backbones, while reporting both accuracy and confidence-faithfulness metrics, would strengthen the generalization claims.
  3. The contribution of PRG to confidence faithfulness is not directly established. Table 7 evaluates the PRG ablations only through accuracy, while Tables 2 and 9 assess confidence faithfulness only on CyberMetric-500. Reporting confidence-faithfulness metrics for the existing ablation checkpoints and financial models would clarify PRG’s specific benefit and its cross-domain applicability.
  4. The posterior-entropy justification drops policy-dependent terms without sufficient justification. Appendix A treats the normalization factor and Epθ[Pslog⁡Ps]\mathbb{E}_{p_\theta}[P_s\log P_s] as constants, although they generally vary with the policy. The derivation should retain their gradient contributions or clearly qualify the weighted objective as a heuristic surrogate.
Questions for authors

My main concern is the experimental validation. I would consider raising my score if the authors provide better-controlled baseline comparisons and demonstrate gains in both accuracy and confidence faithfulness against a strong baseline on additional backbones. See the Critical Weaknesses above for details.

Reviewer_D3vROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HyTuning, a hybrid post-training framework combining reasoning distillation with reinforcement learning from internal feedback under limited supervision. Its key signal, Progressive Reasoning Gain (PRG), adjusts their relative weights based on confidence changes along reasoning traces. Experiments show accuracy improvements across cybersecurity and financial tasks, with confidence-faithfulness evaluation on CyberMetric-500.

Critical strengths

1, HyTuning achieves an average accuracy of 0.541 versus 0.494 for the strongest baseline under partial supervision, with further accuracy gains demonstrated across model families and financial tasks. 2, Using SAR, which is not directly optimized during training, alongside a blinded expert study provides useful preliminary support for the confidence-related motivation.

Critical weaknesses

1, Appendix A drops normalization and correction terms that generally remain parameter-dependent, even with detached PRG weights. Bounded weights do not justify ignoring their gradients. Consequently, the claimed posterior-entropy interpretation is not established by the current derivation. 2, The fixed-weight ablation changes both adaptivity and the average RD–RLIF balance, while Table 7 reports only accuracy. It therefore does not isolate PRG’s contribution to confidence faithfulness. Controls matching the average weights, ideally supplemented by shuffled PRG weights, should be evaluated using the reported confidence metrics. 3, Describing confidence accumulation as necessary for sound reasoning is too strong: a model already confident in the correct answer can reason correctly without further confidence gains. Appendix C.1 controls for trace length but not initial answer confidence, leaving this potential confound in PRG’s diagnostic value unresolved. 4, Section 3.2 does not sufficiently specify the pseudo-target construction for dummy queries. Since low PRG increases their RD weight, the target-generation procedure directly determines this branch’s learning signal. A precise specification is needed to make the update interpretable and reproducible.

Questions for authors

1, How should the posterior-entropy derivation in Appendix A account for the omitted parameter-dependent terms? 2, Does PRG improve confidence faithfulness over controls with matched average weights or shuffled PRG weights? 3, Does PRG remain predictive of reasoning quality after controlling for initial answer confidence? 4, How are the pseudo-target tokens for dummy queries generated and incorporated into the RD loss?

Reviewer_NA7BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets confidence faithfulness: in high-stakes domains, a model's most confident answers should be correct. It frames this as a selective-prediction property. The method, HyTuning, mixes reasoning distillation on 100 verified traces with self-certainty RL (INTUITOR-style) on unlabeled synthetic queries. The two are weighted per trajectory by Progressive Reasoning Gain (PRG), the average positive increase in answer log-probability across reasoning steps. On Qwen2.5-3B, across three cybersecurity MCQ benchmarks plus MMLU, HyTuning reaches 0.541 average accuracy under partial supervision, vs. 0.494 for INTUITOR. It roughly matches fully supervised GRPO (0.567 vs. 0.570) and gives the best selective accuracy on CyberMetric. Further experiments cover ablations, other backbones, a finance domain, and a four-expert study of PRG.

Critical strengths

S1. A practical recipe for label-scarce domains. With only 100 verified traces, Qwen2.5-3B rises from 0.398 to 0.541 average accuracy (Table 1), and the gain carries over to finance (0.472 vs. 0.401, Table 4).

S2. Data-matched fusion ablations. On identical data, equal weighting (0.489) and sequential training (0.477 / 0.498) all trail HyTuning (0.541, Table 7). This shows that how the two signals are combined matters.

S3. A careful expert study of PRG. Four experts rated the traces without seeing their PRG scores, and found the lowest-PRG bin significantly weaker (p=0.013 / 0.006, Table 6). The authors also report that PRG does not separate the higher bins, which supports the one-sided design.

Critical weaknesses

W1. The contribution is incremental. RD and RLIF are both common practice. Combining them, though with a good recipe, is not very novel. There are other papers regarding such combinations and some are not cited.

  • Jianhao Yan et al. Learning to Reason under Off-Policy Guidance. NeurIPS, 2025.
  • Wenhao Zhang et al. On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting. ICLR, 2026.
  • Yuxin Zuo et al. TTRL: Test-Time Reinforcement Learning. NeurIPS, 2025.

W2. The comparisons would benefit from tighter controls.** In Table 1, each baseline sees either the traces or the unlabeled queries, while HyTuning sees both. Several baselines also fall below the untuned backbone (e.g., SFT 0.188 vs. 0.398), and all results are single runs. Multiple seeds and a data-matched hybrid baseline would make the gains more secure.

W3. Writing needs to be improved. Intro didn't give clear intuition and contributions. Sec 3 quite hard to understand

Questions for authors

Address questions in weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.