acceptodds
Under review as a conference paper at ICLR 2027

Knowing When to Defer: Training Language Models to Generate an Answer and Its Confidence Level Together for the Thresholds That Matter

Abstract

A language model that reports its confidence level in each answer allows a user to accept the answer when that confidence level meets a threshold, and to defer it otherwise. This threshold depends on the relative costs of a deferral and of a wrong acceptance. Applications usually face only a limited range of such costs, which gives a *band* of meaningful thresholds. Previous work trains models under the Brier score, which ignores this and weights every threshold equally, so some of the training signal goes to confidence values on which no decision depends. The clipped Brier score weights only the threshold band, but cannot distinguish confidence reports on the same side outside it, and thresholds that may come into use later are not weighted either. We ask how to prioritise current decisions while accounting for learning behaviour and future adaptation. Our approach is the first built to tackle this aim. We first propose a reward that combines a binary correctness score with a weighted confidence score. Our confidence score mixes the clipped Brier score with a small full-range Brier term, the *reserve*, in a controllable proportion that we call the *reserve weight*. For a fixed answer, any positive reserve weight makes the true probability of correctness the unique minimiser of the expected score, so the band concentrates the learning signal without moving its target. We then use this score for three different training systems that each allow a confidence level to be reported for a model's generated answer: a confidence reader trained on a frozen model's features; a separate model trained to report confidence for fixed answers; and a model trained to answer and report confidence jointly by reinforcement learning. Our analysis shows how the band, the reserve weight and the score's weight against the correctness reward set the incentive to answer correctly, giving a rule for placing and weighting the band. With the answer model fixed and only the confidence trained, our score gives the best-calibrated confidence among all previous methods we evaluated, and lowers expected calibration error to less than half that of a model trained under the full-range Brier reward (0.030 against 0.067 on HotpotQA, 0.034 against 0.093 on MedQA). In joint training on MedQA, our score not only cuts *decision cost* at the targeted band against the full Brier reward, but also shows better calibration and ranking over the whole range. These tests support one theory to place the band where the model's confidence lies and weight it as strongly as the incentive to answer correctly allows, which keeps the model focused on the thresholds that matter.

Primary area:
foundation or frontier models, including LLMs
Keywords:
confidence calibration, uncertainty quantification, large language models, reinforcement learning, decision-making under uncertainty, hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vm9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the gap between standard LLM confidence calibration and how confidence is actually used in real decisions. Real-world deployments usually operate within a band of decision thresholds, but full Brier training wastes most signal on irrelevant thresholds, while clipped Brier cannot tell confidence values apart outside the band. The authors propose a weighted scoring rule that mixes band-focused clipped Brier with a small full-range Brier "reserve" term. They formally prove the score keeps the incentive to answer correctly and is strictly proper. Tested across three training setups (frozen feature reader, separate confidence reporter, full joint RL) on two QA datasets, the method roughly halves calibration error and lowers decision cost within the target threshold band.

Critical strengths
  1. Clearly motivated problem with direct real-world relevance
  2. Solid theoretical grounding with clean proofs of key properties
  3. Full reproducibility details included in the appendix
Critical weaknesses
  1. It is hard to tell how much of the gain comes from the proposed scoring rule itself. The idea is interesting, but I am not sure the current experiments are enough to show that the proposed scoring rule is the key factor behind the improvements. It would be useful to see how it compares with simpler alternatives, such as standard Brier score training or other threshold-aware objectives.
  2. The evaluation is quite narrow. All experiments use the same 4B model and only two QA datasets. I wonder if the results would hold for larger models or tasks beyond QA. At the moment, it is difficult to tell how general the method really is.
  3. The choice of (\eta) seems somewhat arbitrary. The paper uses (\eta=0.1), but does not make it clear why this value is chosen or how much the results depend on it. Does the method still work well with different values of (\eta), or does it require careful tuning for each setting? This matters for anyone trying to use the method in practice.
  4. The role of the reserve term is not fully clear. The main idea is to combine the clipped Brier score with a full-range Brier term, so that the model focuses on the target threshold band without losing calibration outside it. However, it is not clear how much the reserve term actually helps in practice, beyond ensuring strict propriety theoretically. Could a simpler alternative achieve a similar trade-off between focusing on the target band and maintaining calibration outside it?
Questions for authors

See weaknesses.

Reviewer_z7C1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to report a confidence level alongside the answer of a language model, for the case where the deferral cost lies in a known range that maps to a band of decision thresholds. The authors propose a score combining the clipped Brier score on that band with a small full-range Brier term, and use it to train a confidence reader, a separate reporter, or the answer and confidence jointly. Experiments use Qwen3-4B-Instruct on HotpotQA and MedQA, with RLCR and a few classical calibration methods as baselines.

Critical strengths
  • The writing is mostly clear.
  • The score comes with a theoretical development.
  • The method can be cheap to apply. The band reader is fitted in minutes on frozen features and requires no retraining of the LLM.
Critical weaknesses
  • The scope and impact of the contribution is overly narrow. The proposed approach is limited to calibration when the cost is defined as a range, with an asymmetry between accepting and deferring. The paper poorly motivates why this specific case is significant, and the main motivation is a single illustrative clinical example. The method is further restricted to QA settings only. The applications at the intersection of these constraints appear severely limited, and the potential impact appears tiny.
  • Important baselines and ablations are missing. The closest related methods are discussed in Sections 3 and 5 but not compared against, namely the truncated Beta weight, C3RL, Deliberative Searcher and RLSR. Standard post-hoc calibration methods such as temperature scaling and isotonic regression are also absent. An ablation on the reserve is missing. In particular, I did not see results for the pure clipped Brier score (η = 0), which would be the most direct way to show that the reserve matters.
  • The experimental setting is weak. A single model (Qwen3-4B) is evaluated on two benchmarks. Calibration methods are trained on HotpotQA or MedQA and evaluated in distribution only. Evaluation out of distribution is missing, although this is where calibration typically degrades. Table 2 reports means over three seeds without confidence intervals or standard deviations.
  • The practical significance of the results in Sections 4.2 and 4.3 is unclear. The paper fails to explain clearly the motivation and the conclusions to gain from these experiments.
  • The writing needs to be improved in some sections. For example, the abstract is hard to understand by itself and is not self-contained. The paper relies heavily on its own vocabulary (path, arm, reader, reporter, reserve, limit setting, equal-weight setting, cut share, gates), which makes it hard to follow.
Questions for authors
  1. The paper does not seem to specify the training set used for GRPO. In particular, is it disjoint from the test set? Appendix C states that the 2,000 reader-fitting questions may overlap with those used to train the answerer, but the training split itself is not described.
  2. Does the paper assume that the cost is uniformly distributed within the band? The weighted score puts uniform weight on thresholds inside the band, which corresponds to a uniform prior over deferral costs.
Reviewer_ZpepOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies confidence estimation for accept-or-defer decisions when deployment thresholds lie in a selected band. It proposes a reward combining the clipped Brier score over that band with a small full-range Brier reserve, and applies it to frozen-feature readers, fixed-answer reporters, and joint answer-confidence training with GRPO. The authors derive conditions for strict propriety and preserving correctness incentives. Experiments on Qwen3-4B with HotpotQA and MedQA show improved calibration in some fixed-answer settings and lower decision cost for some MedQA joint-training bands, alongside accuracy losses in other settings.

Critical strengths
  • The connection between proper scoring rules and threshold-averaged decision regret provides a clear framework for studying decision-aware confidence.

  • The two propositions state interpretable conditions for retaining a truthful confidence target and an incentive for correct answers. The same score is evaluated in three training settings.

  • The paper reports calibration, ranking, accuracy, and band-wise decision costs. The fixed-answer reader and reporter show encouraging ECE results.

Critical weaknesses
  • The novelty is limited. The objective is a mixture of the existing clipped Brier score and standard Brier score, with theoretical properties closely related to established proper-scoring-rule results. The contribution beyond applying this combination to LLM training is unclear.

  • The training bands seem insufficiently motivated. The three ranges are not derived from actual deployment costs for HotpotQA or MedQA, and the method presumes that future cost ranges are available during training. Section 4.4 even recommends placing the training band where model confidence lies, which complicates the proposed deployment-cost motivation.

  • The need for band-specific confidence training is unconvincing. Full-Brier baselines are already evaluated at multiple deployment thresholds, and a single calibrated model can adapt its acceptance threshold to application- or question-specific costs. The results do not show a consistent enough advantage to justify committing to a band during training.

  • Joint-training gains are inconsistent. All three HotpotQA variants have higher mean decision cost than RLCR at their respective training bands; the high-band model also reduces accuracy from 0.562 to 0.467. On MedQA, the high-band improvement in decision cost comes with a 2.5-point accuracy drop.

  • The ablations do not isolate what drives the improvement. Band changes also change the maximum confidence-reward weight, while the equal-weight setting changes both lambda and the reserve weight. Evaluation on one 4B model and two QA datasets further limits the generality of the findings.

Questions for authors
  • What real deployment setting provides a stable threshold band before training? How would the method handle costs that vary across users, questions, or time?

  • Could you evaluate a single general-purpose calibrated model under per-question costs and shifts in deployment bands, and quantify the benefit of band-specific training over this flexible approach?

  • Can the effects of band location, reward weight, and reserve weight be isolated through matched-weight and parameter-sweep experiments?

  • What explains the lack of joint-training gains on HotpotQA and the accuracy losses? Do these trends persist across model scales and additional datasets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.