acceptodds
Under review as a conference paper at ICLR 2027

Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation

12 views

Abstract

Modern language models fail a fundamental requirement of trustworthy intelligence: knowing when not to answer. Despite achieving impressive accuracy on benchmarks, these models produce confident hallucinations, even when wrong answers carry catastrophic consequences. Our evaluations on GSM8K, MedQA and GPQA show frontier models almost never abstain despite explicit warnings of severe penalties, suggesting that prompts cannot override training that rewards any answer over no answer. As a remedy, we propose Reinforced Hesitation (RH): a modification to Reinforcement Learning from Verifiable Rewards (RLVR) to use ternary rewards (+1 correct, 0 abstention, - error) instead of binary. Controlled experiments on logic puzzles reveal that varying produces distinct models along a Pareto frontier, where each training penalty yields the optimal model for its corresponding risk regime: low penalties produce aggressive answerers, high penalties conservative abstainers. The same frontier holds on MATH Levels 4–5 and on medical QA, where it transfers to an unseen dataset. We then introduce two inference strategies that exploit trained abstention as a coordination signal: cascading routes queries through models with decreasing risk tolerance, while self-cascading re-queries the same model on abstention. Both outperform majority voting with lower computational cost. These results establish abstention as a first-class training objective that transforms "I don't know" from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Hesitation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bpXmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies selective answering when incorrect responses incur an explicit cost. It proposes Reinforced Hesitation (RH), which uses a ternary reward of +1 for correct answers, 0 for abstention, and −λ for incorrect answers. Training with different penalties produces models with different coverage–error trade-offs. The paper evaluates this behavior on Knights & Knaves, MATH, and medical QA, including transfer from MedMCQA to MedQA. It also introduces cascading across models trained at different penalties and self-cascading through repeated sampling after abstention. These strategies use learned abstention to allocate inference computation and obtain competitive accuracy with fewer model calls than fixed-budget majority voting.

Critical strengths
  1. The motivation is compelling and practically meaningful. The paper addresses an important mismatch between accuracy-oriented training and deployment settings where incorrect answers can be substantially more costly than abstentions. I appreciate the explicit treatment of this trade-off and the evaluation across reasoning and knowledge-based tasks. The targeting analysis in Appendix I also provides useful evidence that learned abstentions concentrate on problems where the baseline is more likely to fail.

  2. Using abstention to drive adaptive inference is the most appealing contribution. Cascading and self-cascading give learned abstention a concrete computational role. The MATH cascade reaches 89.97% correct with 1.58 average model calls, compared with 91.05% for 16-sample majority voting. Although this does not establish uniform dominance, it is a promising efficiency trade-off. This part of the paper substantially strengthens my interest in the work beyond the reward modification alone.

Critical weaknesses
  1. The algorithmic novelty of the training objective is limited relative to closely related work. At λ = 1, RH's content reward coincides with TruthRL's default ternary reward. TruthRL also investigates alternative abstention rewards (Table 5 of its June 2026 version), and the submission acknowledges the relationship to reward sweeps in Jha et al. Thus, both explicit abstention training and adjusting the reward to change the coverage–error trade-off have close precedents. I view the broader risk-specialist analysis and the inference strategies as the main potential contributions; the reward formulation itself provides limited additional novelty.

  2. Supporting multiple risk preferences introduces a substantial training and deployment burden. The reported specialist frontier relies on separately trained checkpoints for different penalties, while the prompt-conditioned model in Section 3.5 does not match that frontier. A deployment with one fixed risk preference can use a single specialist, and self-cascading also uses one model. However, supporting several risk preferences or the heterogeneous cascade requires maintaining multiple checkpoints, with additional training, storage, and serving requirements. The RLCR comparison in Appendix E.1, Table 6 provides encouraging evidence of higher utility for RH at nonzero penalties, but these performance gains come with reduced flexibility in changing the risk preference. This remains a practical limitation of the proposed specialist approach.

Questions for authors

Deployment clarification: Could you clarify the aggregate training, storage, and serving costs of the specialist family, and how a deployment should select among existing specialists when its error cost changes? This would help assess the practicality of the proposed approach without requiring a new training algorithm.

1 comment
AuthorsOpenReview ↗

Author's response

Thank you for the encouraging review. We agree that the reward formulation has close precedents (Appendix A). Our contribution lies in analyzing risk-specialized models and using abstention at inference.

Training and serving costs. On MATH, each 4B specialist takes 37.7–49.8 GPU-hours to train. All five total 214.0 GPU-hours versus 34.5 for RLCR, or roughly 6×, excluding the shared format warmup used by the specialists. The runs used both H100 and B200 GPUs, so this ratio is indicative. Each bf16 checkpoint occupies about 9.3 GB (46.5 GB for five).

For a fixed deployment error cost, only one specialist needs to be served, at the cost of any single model; self-cascading reuses that model. Only a heterogeneous cascade needs multiple checkpoints available. The depth-four cascade averages 1.53 calls and generates 7.6k characters per problem, compared with 64.5k for 16-sample majority voting. We have not measured latency or peak serving memory.

How many specialists? Fewer than five appear necessary. Adjacent models are often statistically tied: at deployment penalty 5, λ=1, 2, and 5 are indistinguishable in paired tests. The coarse family {0, 1, 10} contains a best or tied-best model at each evaluated penalty on MATH, so three training runs may suffice for those settings.

Choosing a model when costs change. Retraining is not needed. For each specialist, utility is linear in the deployment penalty, U(λ) = P(correct) − λP(wrong). One evaluation pass on development data defines each line; selecting the highest line gives the best estimated specialist at any new cost. On MATH validation problems, this upper envelope switches from the λ=0 specialist to λ=1 at a cost of about 0.17, to λ=5 near 5.7, and to λ=10 above 7.6. Thus the three-model grid covers the evaluated penalties, while finer cost ranges may justify retaining λ=5.

RLCR trades some utility for one adjustable checkpoint. At every nonzero evaluated cost, the matched specialist beats thresholded RLCR by +0.015 to +0.186, with paired confidence intervals excluding zero (Appendix E.1, Table 6). For a known cost, deploy the corresponding upper-envelope specialist; for varying costs, retain a small family and switch models. If one checkpoint is a hard constraint, thresholding RLCR remains a viable alternative with the measured utility loss.

Reviewer_YhJCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether language models can learn to abstain when incorrect answers are costly. A prompting study across eleven models finds limited sensitivity to stated error penalties. The proposed Reinforced Hesitation uses an RLVR content reward of +1 for correct answers, 0 for abstention, and −λ for incorrect answers. Fixed-penalty specialists are trained on Knights & Knaves, MATH Levels 4-5, and medical QA. The paper evaluates their coverage/error trade-offs, a penalty-conditioned model, cross-specialist cascading, and repeated sampling of a single specialist until it provides a non-abstaining answer. The main claims concern risk-specialized abstention and the use of abstention signals to allocate inference computation.

Critical strengths
  1. The experiments provide evidence of selective abstention and transfer across datasets. On MATH, Table 4 reports that increasing λ from 0 to 10 reduces the wrong-answer rate from 11.44% to 2.75%, while abstention rises from 0.08% to 34.39%. Appendix I shows that the λ = 0 model performs worse on problems declined by moderate-penalty specialists than on the full dataset, supporting targeted abstention. The MedMCQA-to-MedQA evaluation in Appendix H provides additional evidence that the behavior transfers to a different question distribution.
  2. The comparisons and negative results help define the scope of the method. Table 6 compares RH with an RLCR-style confidence model using deployment-dependent thresholds. The dynamic-penalty experiment shows that a single conditioned model does not recover the specialist frontier, and the MATH inference discussion acknowledges the higher raw accuracy of full majority voting. These results are useful for assessing when direct risk-specialized training might offer practical value.
Critical weaknesses
  1. The methodological innovation in this paper is limited, and the added value provided by the specialist family has not been sufficiently justified. TruthRL (Wei et al., ICML 2026) already employs the GRPO algorithm and incorporates a reward mechanism, +1 for correct, 0 for abstention, and -1 for incorrect, while also exploring various settings for the abstention reward. Appendix A of this paper acknowledges that the reward mechanism studied by Jha et al. (2026), under the condition λ = (1 + r_abs)/(1 − r_abs), is essentially equivalent to the selective decision-making problem addressed here. Consequently, the use of a neutral abstention reward and the establishment of decision thresholds for answering versus abstaining based on rewards are already established concepts. While conducting a broader sweep of penalty parameters may yield useful empirical insights, the paper needs to further clarify what specific additional insights or mechanisms exist that distinguish this approach from a mere extension of existing abstention training methods.
  2. The ideal approach is complementary specialization, which can improve cascading effects. However, the data in Sections 4.1–4.2 show that on the MATH dataset, a self-cascading model with a single expert (λ = 1) achieves an accuracy of 90.30% with an average of 1.43 queries, whereas a five-expert cascading model yields an accuracy of 89.97% with an average of 1.58 queries. This comparison does not demonstrate superiority across all risk levels, nor does the text address the uncertainty associated with these results.
  3. These key empirical comparisons lack independent testing and evaluation, and they have not been subjected to sufficiently detailed cost accounting. Appendix B.3 notes that the results for MATH were obtained using a validation set of 2,538 problems, the same dataset employed for checkpoint selection. This introduces the risk of selection bias in the reported comparisons between the "expert model" and the baseline models.
  4. The number of queries is, at best, an incomplete metric for gauging computational cost. Figure 16 illustrates the variations in response length across different expert models. Furthermore, the assumptions regarding the serving of multi-checkpoint cascades are not sufficiently specified to allow for an assessment of their associated memory and latency costs. The factor of a terminal risk trade-off further undermines the paper’s claim that both strategies outperform majority voting while incurring lower computational costs.
Questions for authors
  1. Relative to published ternary-reward abstention training such as TruthRL, what specific new mechanism or empirical finding is established by RH? Which result isolates the value of risk specialization beyond obtaining different operating points through a reward sweep?
  2. How do the cross-specialist cascade, the best single-specialist self-cascade, and an RLCR-based adaptive policy compare at matched terminal error rate or deployment utility? In particular, what benefit does the five-model family provide over the MATH λ = 1 self-cascade once uncertainty in the reported correctness and query counts is considered?
  3. Can the principal MATH comparisons be evaluated on an untouched test split after checkpoints and decision rules are fixed on a development split? Please report question-level uncertainty for the relative gains and clarify how many independently trained runs support the comparisons.
  4. Can you report total generated tokens, measured latency, and the serving assumptions for each method, alongside expected deployment utility? A comparison with confidence-based or agreement-based early stopping at a matched budget would clarify whether learned abstention offers an advantage beyond sequential sampling itself.
1 comment
AuthorsOpenReview ↗

Author's response

Thank you for the detailed review. We address your questions using paired bootstrap analyses of existing MATH rollouts (2,538 problems; 10,000 resamples), counting format errors as wrong. Utility is correct − λ·wrong.

Q1. Novelty and evidence for specialization. The reward formulation is not new: at λ=1 it matches TruthRL, and Jha et al.'s sweep is equivalent (Appendix A). TruthRL chooses one model under a fixed score. We instead train for a deployment error cost and test whether this improves decisions at that cost.

The cleanest comparison is RLCR, which obtains all its operating points by thresholding one calibrated model. Against RLCR thresholded at λ/(1+λ), matched specialists gain +0.015 [0.007, 0.023], +0.044 [0.031, 0.058], +0.040 [0.018, 0.062], and +0.186 [0.142, 0.229] at penalties 1, 2, 5, and 10; they tie at 0. A prompt-conditioned model does not recover this frontier (Section 3.5); abstention targets difficult questions (Appendix I) and transfers from MedMCQA to held-out MedQA (Appendix H). Q4 isolates the training benefit from the sampling rule.

Q2. Matched comparisons on MATH.

MethodQueriesλ=0λ=1λ=2λ=5
Majority vote, λ=0160.9100.8220.7330.467
RLCR + threshold10.8870.7990.6930.517
λ=1 specialist10.8760.8140.7510.563
Self-cascade, λ=11.430.9030.8240.7450.509
Cascade 10→5→2→11.530.8850.8170.7480.542
Full cascade1.580.9000.8000.7010.402

Relative to λ=1 self-cascading, the depth-four cascade changes utility by −0.018 [−0.023, −0.013], −0.008 [−0.015, 0.000], +0.003 [−0.009, 0.015], and +0.033 [0.009, 0.059] at penalties 0, 1, 2, and 5. The heterogeneous family helps over self-cascading only when errors are costly. The full cascade (which reaches λ=0) is significantly worse than majority voting at every evaluated penalty. We will remove the abstract's blanket cascade claim and describe depth as a risk parameter. At penalty 5, a single λ=1 query is best among methods using at most 1.6 queries.

Q3. Held-out evaluation and runs. Each MATH and Knights & Knaves specialist is one run. In the medical suite, λ ∈ {1, 2, 5} have two seeds: conditional accuracy varies by 0.6–3.8 points across seeds, compared with 9.0–18.7 between adjacent penalties. We agree that choosing MATH checkpoints on the same 2,538 problems used for reporting is a limitation; we cannot supply an untouched MATH split in this rebuttal. The medical suite follows the suggested protocol: every run uses step 150, with no selection, and MedQA is never used for decisions. The frontier and targeted abstention persist there (Table 10, Appendix I). In the final version, we will also report all MATH models at one common step fixed in advance.

Q4. Compute and early stopping. Generated characters per problem are 4.0–4.6k for one query, 6.9k for self-cascading, 7.6k for depth four, 8.0k for the full cascade, and 64.5k for 16-sample majority voting. Abstentions are longer than answers (7.1k versus 3.8k at λ=1), so the saving is about 8–9× in generated text, not 10–11× as query counts suggest. We will add token counts. We have not measured latency: cascades average 1.43–1.58 sequential calls, while the 16 majority-vote samples can run in parallel at 16× compute. Each 4B model occupies about 9.3 GB in bf16.

We also ran your proposed agreement-based stopping rule (sample until an answer repeats, at most 16 times). At comparable budgets (2.07 versus 2.14 queries), the λ=1 specialist beats the λ=0 model by +0.022 [0.011, 0.033], +0.063 [0.046, 0.082], and +0.187 [0.148, 0.228] at penalties 1, 2, and 5; it loses by 0.019 at penalty 0. Using about one-eighth as many queries, it also beats 16-sample majority voting at penalties 2 and 5 (+0.043 [0.023, 0.063] and +0.146 [0.103, 0.190]). Since both models use the same stopping rule, the difference comes from risk-sensitive training.

Reviewer_f3KMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper applies RLVR with a ternary reward and sweeps the penalty for wrong answers.

Critical strengths
  1. Results from sweeping the error penalty in the ternary reward.
  2. A consistent, targeted trend across three domains. A higher λ gives fewer errors and higher conditional accuracy.
Critical weaknesses
  1. The main claim, that training at each penalty yields the best model for that penalty, is not supported by the paper's own results.
  • Some models are never the best. On Knights & Knaves (Fig. 1), the λ=2 model is not the best at any penalty: λ=1 or λ=5 always scores at least 1.5 points higher. The λ=20 model abstains on every question. On MATH, λ=1 beats λ=2 at penalties 1, 2 and 5 (Table 4). On medical QA, λ=2 beats λ=1 at penalty 1 (Table 10).
  • The differences are within noise. Each model is trained only once. On MATH, the disputed gaps (0.4 and 1.0 points) are smaller than the noise in a single result (1.5 and 2.8 points). The checkpoints were also chosen on the evaluation set, which favours each model at its own penalty.
  • What the results do show is expected. A higher penalty gives fewer answers and fewer errors, which follows from simple decision theory (answer only when confidence exceeds λ/(1+λ)). Godin et al. (arXiv 1902.10236) already showed this with the same three-way reward.
  1. The main findings have been reported in existing work. Godin et al. (arXiv 1902.10236) and TruthRL (arXiv 2509.25760) have already established that:
  • A ternary RL reward makes a model abstain selectively, with fewer errors and higher precision. Godin et al. showed this for a non-LLM agent, and TruthRL for LLMs at λ=1.
  • Raising the penalty trades answer rate for precision (Godin et al.'s reward sweep).
  • Binary rewards drive abstention toward zero (TruthRL's ablation).
  1. When a high-penalty model abstains, it is unclear why one would want a lower-penalty model to answer instead, since it is likely to guess or hallucinate. This seems to undo the abstention that the high-penalty training is designed to produce.
Questions for authors
  1. What findings does this work offer beyond the existing work (arXiv 1902.10236 and arXiv 2509.25760)?
  2. When a high-penalty model abstains, why pass the question to a lower-penalty model, which is likely to guess or hallucinate? This seems to undo the abstention that the high-penalty training is designed to produce. Could the authors explain the motivation for this design?
1 comment
AuthorsOpenReview ↗

Author's response

Thank you for the careful review. We agree that our headline optimality claim is too strong and will correct it.

1. Optimality and what the results support. Your counterexamples hold. Paired bootstrap tests on MATH (2,538 problems; 10,000 resamples) confirm that λ=1 beats λ=2 at deployment penalty 2 (+0.014 [0.004, 0.025]), while λ=1, 2, and 5 are statistically tied at penalty 5 (λ=1 minus λ=5: +0.007 [−0.013, 0.026]). We will remove the claim that each training penalty yields the optimal model at its matching deployment penalty from the abstract, Figure 1, and Section 3.

The narrower finding remains meaningful: no model is best across risk levels. The λ=0 model beats λ=1 when errors are free (+0.009 [0.004, 0.014]), while λ=10 beats λ=1 at penalty 10 (+0.103 [0.040, 0.165]). Relative to λ=0, matched specialists gain +0.043, +0.081, +0.244, and +0.612 at penalties 1, 2, 5, and 10 (all paired intervals exclude zero). A coarse set, λ ∈ {0, 1, 10}, contains a best or tied-best model at every evaluated penalty. We will also label λ=20's collapse to universal abstention as a failure mode.

2. Statistical uncertainty and selection. The models are tested on the same questions, so paired intervals are more informative than comparing their individual standard errors. MATH specialists are single runs. For the medical suite, we trained second seeds at λ ∈ {1, 2, 5}: conditional accuracy varies by 0.6–3.8 points across seeds, versus 9.0–18.7 points between adjacent penalties. All six runs exceed their Bayes thresholds, λ/(1+λ), on both medical datasets. We acknowledge that MATH checkpoints were selected using evaluation problems. In contrast, every medical run uses step 150 without checkpoint selection, and the ordering persists on MedQA, which was never used for any decision (Table 10).

3. Beyond decision theory. We agree that the Bayes rule is to answer when confidence is at least λ/(1+λ). This is our design principle, not an empirical finding. The question is whether training actually produces good decisions under these costs. Prompting frontier models rarely induces abstention even at high stated penalties (Section 2). More importantly, a specialist trained for the deployment cost outperforms thresholding a calibrated RLCR model at that same Bayes threshold. On MATH, RH minus RLCR is +0.015 [0.007, 0.023], +0.044 [0.031, 0.058], +0.040 [0.018, 0.062], and +0.186 [0.142, 0.229] at λ=1, 2, 5, and 10; they tie at λ=0. A single prompt-conditioned model also fails to recover the frontier (Section 3.5). Abstention is targeted: on questions declined by λ=1, the λ=0 model is correct only 33.6% of the time, versus 88.5% overall (Appendix I).

4. Godin et al. and TruthRL. We will cite Godin et al. Their knowledge-graph QA agent uses rewards (+10, 0, −0.1), equivalent to λ=0.01, and sweeps up to λ=0.05 before selecting one model by QA score. We study LLM reasoning at substantially higher error costs (confidence thresholds 50–91%). TruthRL trains at our λ=1. Its Table 5 changes the abstention reward (−0.5, 0, +0.5), equivalent to λ≈0.33, 1, and 3. It calls +0.5 over-abstention, although stronger abstention is appropriate when deployment errors cost λ=3. The difference is our evaluation perspective: we treat λ as the deployment cost, train a family of specialists, and use abstention for routing. TruthRL does not vary the error penalty, train such a family, or route at inference. Its binary-reward ablation agrees with our λ=0 result (0.08% abstention on MATH).

5. Cascading and residual risk. Passing a declined question to a lower-penalty model does not necessarily force a guess: every tier with λ>0 may still abstain. But you are right that reaching λ=0 removes this protection. Paired tests show the full cascade is significantly worse than 16-sample majority voting at every evaluated penalty on MATH and both medical datasets. We will remove the unconditional cascade claim and treat depth as a risk parameter. Stopping at λ=1 (10→5→2→1) matches majority voting at penalties 1 and 2 and beats it at penalty 5 (+0.075 [0.033, 0.119]) with about one-tenth as many queries. At penalty 5, a single λ=1 query does better still (0.563 versus 0.542). We therefore recommend stopping before λ=0 unless errors are nearly free.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.