acceptodds
Withdrawn from ICLR 2027

Teaching Language Models to Say What They Know: Reinforcement Learning with Internal Confidence Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As language model adoption increases, AI overconfidence can affect more people, increasing the total harm—the severity of its effects multiplied across the population exposed. Human communication relies in part on hedging; people qualify claims when they are uncertain, helping others judge how much to trust a statement. Perpetual overconfidence breeds mistrust. Language models should therefore express confidence that tracks their internal uncertainty, but standard post-training does not ask them to. We present Reinforcement Learning with Internal Confidence Alignment (RLICA) which utilizes semantic entropy estimates during the post-training process to align internal confidence of the model with a proper scoring rule tied to the probability of language model correctness. Against outcome supervision, RLICA lowers ECE from 0.221 to 0.189 while raising accuracy from .594 to .625 out-of-distribution. Its confidence report is coherent: when the model's own samples disagree it states a confidence equal to its accuracy rather than refusing to answer, and its verbalized confidence is low on 88 percent of the answers whose prose states doubt both in distribution and out, against 45 percent in distribution and 31 percent out under outcome supervision, where the verbalized confidence stays high while the prose hedges. We take this work as promising evidence for the paradigm of supervising verbalized confidence against the model's own internal confidence to produce faithful calibration of language models.

Primary area:
interpretability and explainable AI
Keywords:
calibration, verbalized confidence, reinforcement learning, trustworthy ai, ai safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SjjZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RLICA, which extends RLCR (GRPO with correctness, format and Brier rewards on a verbalized confidence) with a reward that pulls the stated confidence toward an internal estimate of correctness. The estimate is a hidden-state ridge probe for semantic entropy, mapped to a correctness probability by an isotonic fit, both refit periodically on policy rollouts. Training Qwen3.5-4B Base with LoRA on a modified HotpotQA, one run per method, the authors report better out-of-distribution accuracy and calibration than RLCR, and a stated confidence that drops when the model's own reasoning expresses doubt far more often (88% against 31% out of distribution). Abstention, sycophancy, probe-drift and steering analyses follow.

Critical strengths
  1. The paper tests something aggregate calibration cannot see. Asking whether the stated confidence drops when the model's own analysis expresses doubt targets a failure that ECE hides, and the effect here is large rather than marginal. This is the clearest evidence that the alignment term changes how the model reports confidence.

  2. The comparison with RLCR is clean and the reporting is transparent. The two methods share one configuration and differ only in the alignment term, differences come with paired bootstrap intervals, per-dataset results are complete, and a frozen-map ablation shows that the online refit matters. This makes the open questions below easy to locate.

Critical weaknesses
  1. The experiments do not isolate the mechanism, so the claim that the gains come from internal uncertainty is not yet supported. Because the isotonic map is fit on correctness, the internal estimate is in effect a calibrated correctness predictor, and any such predictor could serve as the target; no control keeps a calibrated target while removing the hidden-state information, and only one alignment weight is tried. The claim that the model states a confidence equal to its accuracy when its samples disagree is likewise untested: only aggregate calibration is reported, and the steering experiment uses the frozen base model rather than the trained policy. The proper-scoring-rule framing also overreaches: the combined reward peaks at a weighted average of the correctness probability and the internal estimate, pulling confidence away from the truth where the probe errs, which fits the significant out-of-distribution AUROC drop the abstract omits. The paper's central attribution to hidden states therefore remains open.

  2. The headline gains are small, rest on a single training seed, and are concentrated in individual datasets. Each method is trained once, and the bootstrap intervals resample evaluation prompts, so they capture test-set noise, not run-to-run variation; in distribution, only AUROC differs significantly. Out of distribution, most of the accuracy gain comes from GPQA, where RLCR falls below even RLVR, while SimpleQA alone accounts for more than the entire ECE improvement: without it, RLICA's mean ECE is worse than RLCR's, as on most out-of-distribution sets. GPQA also carries nearly all of the AUROC loss. Inconsistencies in the text, such as 312 versus 208 training steps and comparisons with RLCR labeled as comparisons with RLVR, obscure which runs the tables describe. Robustness across datasets and training runs is therefore not established.

Questions for authors
  1. Can you report RLICA with the internal estimate replaced by the empirical accuracy of the rollouts already sampled for the same prompt, all else fixed, plus a sweep over the alignment weight tracking out-of-distribution AUROC?
  2. What direct evidence supports the claim that stated confidence equals accuracy when samples disagree, for instance a reliability curve on the highest semantic-entropy bins?
  3. Which step count, 312 or 208, produced the evaluated checkpoints, and do the out-of-distribution gains, including RLCR's low GPQA accuracy, recur over at least two more seeds?
  4. Would you narrow the novelty claim in Section 3 to RL with a refit semantic-entropy probe and discuss Ji et al., "Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations" (EMNLP 2025), which finds a linear hidden-state feature governing verbal uncertainty?
Reviewer_ag9VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RLICA, which adds an internal confidence alignment reward to reinforcement learning for verbalized confidence calibration. A semantic entropy probe estimates uncertainty from intermediate hidden states, and its calibrated output supervises the confidence reported by the model. The probe is updated during training. Experiments report improvements in average accuracy, calibration, and consistency between numerical confidence and expressions of doubt.

Critical strengths

The method is reasonable and practically motivated. Using an internal uncertainty estimate to supervise confidence expression is a sensible design, and the reported results show useful gains over RLCR. The analysis of numerical confidence and uncertainty language provides an informative view of how training changes model behavior.

Critical weaknesses

The technical novelty is limited. The method combines an existing semantic entropy probe with an existing calibration objective, with GRPO serving as the optimizer. I feel like this kind of design already existed 2 or 3 years ago and the main contribution of this paper is the new components inside it borrowed from more recent papers. Also the baseline comparison is quite narrow. The experiments include only RLVR and RLCR, omitting established calibration methods. The paper also lacks controls that replace the hidden state probe with other similar methods such as sampling based uncertainty or a correctness probe.

Questions for authors

See above

Reviewer_xqmYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reinforcement Learning with Internal Confidence Alignment (RLICA), a method for improving verbalized confidence calibration in language models. The main idea is to align the model's reported confidence with an internal uncertainty estimate obtained from a semantic entropy probe applied to intermediate hidden states. RLICA augments the standard correctness-based Brier reward with an additional alignment reward encouraging consistency between verbalized and estimated internal confidence. Experiments using Qwen3.5-4B across multiple benchmarks show improvements in accuracy and calibration. The authors also present improved consistency between numerical confidence and linguistic expressions of uncertainty.

Critical strengths
  1. The paper identifies a meaningful limitation of outcome-based confidence calibration: a model's numerical confidence may not align with uncertainty signals reflected in its internal representations or generated language. The proposed approach provides a straightforward way to incorporate hidden-state-based uncertainty estimates into confidence-aware RL. This is a potentially useful direction for improving the reliability of language models.

  2. RLICA improves out-of-distribution accuracy and calibration compared to RLCR, reducing ECE from 0.221 to 0.189 while increasing accuracy from 0.594 to 0.625. The improvement in consistency between verbalized confidence and linguistic expressions of uncertainty is also interesting. In particular, Table 2 shows that when the model expresses doubt in its reasoning, RLICA reports low numerical confidence substantially more frequently than RLCR. These results suggest that the additional alignment objective can influence confidence expression beyond standard outcome-based supervision.

Critical weaknesses
  1. The proposed semantic entropy probe measures answer consistency rather than the probability of correctness. A model can consistently generate incorrect answers while exhibiting low semantic entropy. Therefore, it is unclear whether the proposed estimate faithfully reflects the model's internal confidence.

  2. The paper lacks direct comparisons with simpler alternatives, such as hidden-state correctness probes, token probabilities, or sampling-consistency estimates under the same RL framework. These baselines are important for establishing the advantage of the proposed semantic entropy-based alignment.

  3. The proposed approach primarily adds a squared-error alignment reward to the existing RLCR objective. Similar ideas of aligning verbalized confidence with uncertainty estimates have been explored in prior work, including RLMF and DCA. The main distinction is the use of an intermediate hidden-state probe, whose unique benefits are not sufficiently established.

Questions for authors
  1. How do the authors justify interpreting the calibrated semantic entropy estimate as the model's internal confidence in the correctness of a specific response? Since semantic entropy primarily measures answer consistency, can the authors investigate cases where the model consistently produces incorrect answers and analyze whether the probe remains informative?

  2. Can the authors compare RLICA with an otherwise identical method that uses a hidden-state correctness prediction probe instead of a semantic entropy probe? This would help determine whether the improvements specifically come from semantic entropy-based internal alignment or simply from an additional calibrated confidence signal.

  3. How does RLICA compare with other internal or behavioral uncertainty estimators, such as token probabilities or sampling consistency, when incorporated into the same alignment reward? Such comparisons would help establish the advantage of using intermediate hidden states.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.