acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Signal Uncertainty in LLMs

Abstract

Large language models (LLMs) often produce confident yet incorrect answers, which can lead to risky failures in real-world applications. We study whether post-training can make a model's self-assessment explicit: when the model is uncertain, can it be trained to signal so within its own response? A central design question is where in the response this signal should be exposed: during reasoning, while the answer is still being formed, or at the end, once the answer has been produced. We study both. For end-of-reasoning self-assessment, we train the model to verbalize a confidence score for its response, with the aim of high confidence on correct answers and low confidence on incorrect ones. For during-reasoning self-assessment, we train the model to emit the marker `<uncertain>` as a warning, using final-answer correctness to supervise whether it appears. Across factual question-answering tasks, confidence training improves calibration, and generated warnings identify answers more likely to be incorrect. Both reports support selective retrieval to revise completed answers. We also examine internal representations through confidence-related geometry, confidence-digit probabilities across layers, and probes across model families. On shared responses, these analyses characterize associations between internal states, expressed confidence, and answer correctness.

Primary area:
generative models
Keywords:
Large Language Models, Self Awareness, Uncertainty, Mechanism Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GZzyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies two post-training methods for LLMs to express uncertainty, which are verbalizing a confidence score after the answer and emitting an <uncertain> marker during reasoning. The authors show that confidence training improves calibration and that both uncertainty signals can support downstream tasks, such as selective retrieval to revise completed answers. Internal representations related to the uncertainty signals are further analysed.

Critical strengths

The paper explores a novel idea of generating an explicit uncertainty signal during the reasoning process through post-training.

Critical weaknesses
  1. The paper is hard to follow, and the overall structure should be better organized. For instance, the “Error coverage and precision” section mainly presents experimental results and would be more appropriate in the evaluation section rather than in the method section.
  2. The main experiments are mainly only based on Llama-3-8B-Instruct, which limits the strength of the general conclusions. Although the authors provided other model experiments, but these are mostly single run replications.
  3. Some of the claimed contributions appear too broad and are not sufficiently separated from prior work. For instance, training models to verbalize confidence has already been studied. The paper should more clearly distinguish what has already been established and what is newly contributed here.
  4. In the downstream retrieval task, the retrieval decision is made only after the first response has been completed. Therefore, the experiments do not actually evaluate the practical advantage of emitting the <uncertain> marker during reasoning.
Questions for authors
  1. How were the reward values 5.0, 3.5, and -2.0 in Equation (6) selected? How sensitive are the results to these choices?
  2. When the model emits <uncertain> during reasoning, does the marker itself influence the subsequent reasoning or final answer?
  3. Why is timing of warning not included in the reward function?
Reviewer_DQ2tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how LLMs can explicitly express uncertainty through post-answer confidence scores or reasoning-time <uncertain> markers, both trained using outcome-based rewards. Experiments demonstrate improved confidence calibration, effective error detection, and benefits for selective retrieval. The authors also analyze internal representations associated with uncertainty reporting.

Critical strengths
  1. Effective uncertainty reporting. The proposed training substantially improves confidence calibration (ECE: 52.5% → 4.9%). The learned uncertainty markers also demonstrate strong error-detection performance (81.8% precision and 79.8% recall).
  2. Comprehensive experimental evaluation. The paper includes reward ablations, cross-model experiments, and selective retrieval evaluations. In particular, the shared-answer-candidate comparisons help isolate the effectiveness of uncertainty-based retrieval decisions.
Critical weaknesses
  1. Limited evidence for reasoning-time uncertainty localization. The marker reward depends only on final-answer correctness and marker emission, without supervising its position. Moreover, retrieval occurs after the complete response, leaving the unique benefit of reasoning-time signaling unclear.
  2. Limited downstream gains and evaluation independence. Marker-based retrieval improves EM over No-Ret by only 0.76 points (95% CI: [0.00, 1.56]). Additionally, the overlap between probe development data and the retrieval evaluation pool raises concerns about the robustness of these gains.
Questions for authors

Q1. Reasoning-time localization and intervention.

Does the location of <uncertain> provide predictive information beyond its presence or absence? In particular, can the authors compare the proposed reasoning-time marker against a post-answer binary uncertainty marker under matched training and evaluation conditions? An experiment that actually triggers retrieval or correction when the marker first appears would be especially helpful in establishing the unique value of reasoning-time signaling.

Q2. Independence and robustness of marker-based retrieval.

Could the authors evaluate the marker-based retrieval gate with probe development and evaluation questions completely disjoint? It would also be useful to report the gain over No-Ret across independent training runs and retrieval budgets, with paired uncertainty estimates. This would clarify whether the currently modest downstream improvements are robust.

Q3. Comparisons with simpler confidence-based retrieval gates.

Could the global temperature-scaling and adaptive temperature-scaling baselines be evaluated as retrieval gates using the same first-pass answers, retrieved answer candidates, and retrieval budgets? Such a comparison would help quantify the additional downstream value of reinforcement-learning-based confidence training.

Reviewer_veELOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of LLM confidence calibration on improving model explicit self-assessment with post-training. For end-of-reasoning self-assessment, the authors aim to improve the model’s verbalized confidence; For during-reasoning self-assessment, the authors design an <uncertain> token as a warning. Across several QA tasks (2Wiki, NQ, etc.), the authors report improved model confidence calibration (on ECE). The authors then conduct an analysis of the model's internal representations and show the connection between confidence and model internal states.

Critical strengths
  1. This paper is easy to read, with clear visualizations

  2. The topic of studying the reliability of LLMs with self-reporting (and potentially giving users the warning) is important and timely.

  3. The authors discuss various mechanisms around the proposed <uncertain> tag, including confidence calibration (Section 5.1, Section 5.3), selective retrieval (Section 5.4), and internal representation analysis (Section 6).

Critical weaknesses
  1. The authors propose to study the research question of where the self-assessment should be exposed (in the abstract). However, the question is not truly answered by the experiments. The experiments in the end-of-reasoning setting are basically RLCR (previous work). The experiments in the during-reasoning setting use the <uncertain> tag, which also has limited novelty. The authors did not (1) compare the mechanisms and the correct timing with a unified experimental setting; nor (2) propose a unified mechanism with the collaboration of during and end-of reasoning that shows improved model behavior. To answer the research question, the reviewer would like to suggest that the authors to reconsider the scope of the work well on either one of the aforementioned directions.

  2. The novelty of the work is limited compared to Self-RAG (2023). Although the controlling tag of <uncertain> is not necessarily the same as “retrieve” (reflection tokens) in Self-RAG, the experiments on the downstream application (Section 5.4) are still mainly about selective retrieval. Their differences, e.g., how the judgment certainty differs from the usefulness of retrieval, remain undiscussed.

  3. Even for the trained model, the precision and recall are still low (around 70, Table 1), which suggests a high potential for false positive/negative, which limits the practical utility of the proposed method (and the corresponding supervision pipeline from the final-answer correctness).

Questions for authors
  1. Why is the tag designed to be a single tag instead of a wrapper to show the uncertain field (e.g., <begin of uncertain> and <end of uncertain>), which can potentially make more sense if we want an external checker.

  2. What additional information does the <uncertain> marker provide beyond a thresholded verbalized confidence score?

  3. Some tables (e.g., Table 5) have a mixture of metrics where some are the lower the better (ECE), and some are the higher the better (Acc.). Adding up/down arrows can make the results easier to interpret.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.