acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning Done Right for Explicit Confidence Targets

Abstract

Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliability. Training models to abstain with reinforcement learning is a natural approach, but most existing methods use a fixed error penalty and therefore cannot adapt their behavior to different confidence requirements at inference time. We study explicit confidence targets, where a user specifies a target p in the prompt and the model directly chooses whether to answer or abstain, with the goal of achieving an accuracy of at least p among answered questions. However, learning to respond to different confidence targets with a single model has proven difficult, and prior attempts have failed to distinguish different targets. We show that this failure comes from two problems: the reward signal that distinguishes different values of p is often lost during optimization, while the original error penalty grows without bound as p approaches one. We propose RLEC (Reinforcement Learning with Explicit Confidence Targets), which preserves the learning signal needed to distinguish confidence targets, introduces a stable reward family that keeps the desired answer condition unchanged, and uses reward scheduling to balance effective learning and training stability. Together, these designs enable the first successful training of a single LLM for explicit confidence target control. Extensive experiments across diverse benchmarks and model scales show that RLEC largely preserves general capabilities and achieves competitive calibration performance. RLEC also supports fine-grained target control without an external decision system or threshold calibration after training. By making explicit confidence targets part of the trained policy, RLEC opens a new and more direct path toward behavioral calibration. Our code is available at: [https://anonymous.4open.science/r/RLEC-ICLR-48E5](https://anonymous.4open.science/r/RLEC-ICLR-48E5).

Primary area:
reinforcement learning
Keywords:
LLM, RL, Behavior Confidence

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Hbi8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies explicit confidence targets: a user specifies a target accuracy p in the prompt, and the model itself decides whether to answer or abstain, with the goal that accuracy among answered questions is at least p. Prior work (ERT) attempted this with GRPO but found the model insensitive to p. The paper attributes this failure to two causes: p enters the reward only through the magnitude of the error penalty (−p/(1−p)), yet GRPO's division by the group standard deviation cancels that magnitude in the common case where a rollout group has only two distinct reward values; and the penalty diverges as p→1. The proposed method, RLEC, (i) removes the standard-deviation normalization while keeping mean centering, (ii) multiplies all rewards at a given p by (1−p)^b to bound the penalty without changing the decision rule, and (iii) linearly anneals b from 0 to 1. Experiments on Qwen3-4B-Base and Qwen3-30B-A3B-Base across eight benchmarks compare RLEC with RLVR, TruthRL, ERT, and RLCR.

Critical strengths

The diagnosis points in the right direction. Figure 5(b), which adds the standard deviation back and reproduces the overlapping curves across different values of p, is a convincing ablation — it provides direct causal evidence for the core diagnosis rather than a mere correlation. The statistics in Figure 2 on how the proportions of the three group types evolve during training are a useful new measurement, grounding the "signal loss" argument empirically rather than purely algebraically. In addition, when discussing standard-deviation normalization, Appendix A.1 also brings in work arguing for keeping normalization (Ge et al. 2026; CAL-GRPO) rather than citing selectively, which I appreciate.

Critical weaknesses

1. The methodological contribution is limited, and a key comparison is missing. That dividing by the group standard deviation cancels the reward scale is the same derivation as Dr. GRPO (arXiv:2503.20783), and removing the standard deviation has already been used in several papers; the novelty here is only the observation that the cancelled scale happens to carry p. The claim "unlike all of them" in A.1 also does not hold, since Turtel et al. (2025), cited in the same paragraph, likewise remove normalization to preserve reward magnitude. The (1−p)^b factor and the b schedule lack theoretical grounding: Eq. 19 holds for any positive factor decreasing in p, and nothing justifies why this form, why a linear schedule, or why the endpoint is 1. Moreover, by Eq. 18, b merely scales the advantages uniformly at a fixed p, so the failure of b=1 to separate targets in Section 4.4 looks more like under-training under a fixed budget (b=0.5 separates only late in Figure 5(d)), and a longer budget is never tested. Finally, by the authors' own diagnosis, the most natural fix is to mean-center within the group while switching to batch-level standard deviation; the paper neither discusses nor compares against it — if it works, both the (1−p)^b factor and the b schedule become unnecessary.

2. The experimental evidence does not support the central claim of distinguishing different targets. None of the three metrics in Table 1 measures target responsiveness: Forced Accuracy is measured at p=0, Score measures utility, and AOC penalizes only shortfall, so a model that abstains uniformly conservatively can still achieve a near-zero AOC. Section 4.4 first states that b=1 "fails to distinguish the targets" and then that the three metrics "remain broadly consistent" across b, which shows precisely that these metrics cannot capture this property. The qualitative evidence is also insufficient: Figure 3 shows only three values of p, and on DAPO-Math the 0.1 and 0.5 curves nearly coincide around 50% while 0.9 rises sharply, which looks more like a threshold switch than a graded response; the graded trend in Figure 4(a) is an average over eight benchmarks and may mask per-benchmark step responses; and the only fine-grained single-dataset result (Figure 6) is on AIME24/25, where almost every question should be abstained, making the shape of the response hardest to see. I suggest adding a metric that directly quantifies target differentiation, together with per-benchmark curves over the full grid of p.

Questions for authors

Please see the Weaknesses section.

Reviewer_aZ4JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RLEC (Reinforcement Learning with Explicit Confidence Targets), which trains large language models to decide whether to answer or abstain according to a user-specified confidence target. Compared with GRPO, RLEC retains group mean centering but removes standard deviation normalization, and introduces a scheduled reward-scaling coefficient to moderate penalties at high confidence targets. Experiments with two Qwen3 model scales across eight question-answering benchmarks demonstrate changes in behavior as the target varies and improvements on the reported metrics over several baselines.

Critical strengths

The analysis of GRPO's reward mechanism is interesting. When reward magnitudes encode task conditions, risk preferences, or sample importance, advantage standardization is not necessarily a harmless optimization step.

Critical weaknesses
  • Citation formatting is incorrect throughout the paper.
  • The paper motivates explicit confidence targets as a more direct way to achieve reliable abstention. However, it remains unclear why target-conditioned abstention should be preferred over learning calibrated confidence estimates followed by external thresholding. A well-calibrated confidence estimator can support arbitrary risk thresholds without requiring target-conditioned generation or repeated inference for each target. The claimed advantages of eliminating external decision rules and supporting fine-grained control therefore need stronger empirical or theoretical justification.
  • The evaluation lacks comparisons with several closely related published methods, such as DCPO (ICML 2026), Rewarding Doubt(ICLR 2026), and SAFER (ICLR 2026). RLVR and TruthRL are not explicitly trained to follow varying confidence targets, making them relatively weak baselines for this task. Although ERT provides a direct comparison, the more important question is whether RLEC outperforms calibrated confidence estimation followed by thresholding. The comparison with RLCR is insufficient to establish this advantage, particularly given the mixed results at 4B and the limited confidence resolution observed for the evaluated RLCR model.
  • Although RLEC adjusts its abstention behavior in response to different confidence targets, it does not consistently satisfy the requested accuracy constraints. For example, at p = 0.9, the 4B model achieves only approximately 81% accuracy among answered questions. The authors should distinguish behavioral responsiveness from actual target satisfaction and report additional metrics, such as the maximum target violation and the target satisfaction rate.
Questions for authors

The paper primarily evaluates aggregate abstention rates and accuracy across confidence targets. However, it is unclear whether individual decisions remain consistent as the target changes. Ideally, a model should not abstain on a question at p = 0.5 but answer the same question at p = 0.9. Such monotonicity is naturally ensured by thresholding a fixed confidence score, but is not guaranteed by RLEC. An analysis of sample-level consistency would strengthen the evaluation.

Reviewer_sjqUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies explicit confidence targets, where an LLM receives a target probability in its prompt and directly decides whether to answer or abstain. It identifies a failure mode of group-standardized advantages: for groups containing two distinct reward values, normalization removes their magnitude difference, weakening the direct signal associated with the target. RLEC removes standard-deviation normalization and introduces a target-dependent reward scaling schedule. Experiments on two Qwen3 model scales and eight benchmarks demonstrate target-sensitive abstention while largely preserving forced-answer accuracy.

Critical strengths
  1. The connection between the reward formulation and advantage normalization is concrete and useful. The binary-group calculation makes the proposed failure mechanism transparent, and the normalization ablation supports its practical relevance.

  2. The evaluation distinguishes answer ability, target violations, and utility. Reporting both AOC and Score, together with risk-coverage comparisons, makes the trade-offs more informative than reporting abstention or selective accuracy alone.

Critical weaknesses
  1. Behavioral responsiveness is established more convincingly than reliable target compliance. For example, Section 4.2 reports approximately 81% attempted accuracy at a target of 0.9 for the 4B model, and Table 1 shows substantial remaining AOC on difficult knowledge benchmarks. The limitations acknowledge this, but the central claims should consistently distinguish learning target-dependent behavior from satisfying the requested accuracy.

  2. The evidence for the particular reward schedule is mixed. In Table 3, fixed coefficients achieve better aggregate AOC and Score than the default schedule, while the schedule is selected primarily for forced-answer accuracy and training dynamics. Without uncertainty estimates for these differences, it is difficult to assess how robust this design choice is.

Questions for authors
  1. Could you report per-benchmark target accuracy and coverage with uncertainty intervals, especially at high targets? This would clarify where target compliance holds and where only monotonic adaptation is observed.

  2. How consistent are the differences between the linear schedule and fixed coefficients across training seeds? Does the proposed explanation persist after controlling for the overall update magnitude?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.