- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether LLMs can reliably express uncertainty through natural-language hedging (LC). The authors construct a human-annotated dataset of confidence expressions and develop a lightweight mapper to convert them into numerical confidence scores. Experiments across multiple LLMs and QA benchmarks evaluate calibration and discriminability, showing that explicit hedging prompts (LC+) generally improve over vanilla LC, although performance against existing UQ methods remains mixed. The authors further propose a semantic-uncertainty-guided fine-tuning framework, achieving improvements on SimpleQA and NQ-Open, particularly on the latter, without requiring additional hedging instructions at inference time.
Critical strengths
- The human-annotated dataset and lightweight mapper are valuable contributions. Unlike fixed phrase-level mappings, the mapper handles complete answer sentences. Table 1 reports roughly 72% lower MSE than the best direct-prompt baseline, with substantially lower latency. I think this provides a practical and efficient tool for studying linguistic confidence.
- The systematic evaluation of linguistic confidence is informative. By examining both calibration and discriminability, the paper offers useful insights into spontaneous versus explicitly elicited uncertainty. The fine-tuning results are also encouraging, improving AUROC by about 0.23 and reducing ECE by 0.16 on NQ-Open. Despite mixed results across settings, these findings are worth further investigation.
Critical weaknesses
- The competitiveness of LC+ seems somewhat overstated. Although Tables 2 and Table 3 show improvements over vanilla LC, the results are mixed when compared with existing UQ methods. For instance, even with reasoning enabled, LC+ trails VNC by roughly 0.21 AUROC on GPT-5 (Figure 4, left). This suggests that explicit hedging does not always translate into reliable uncertainty estimation. I would appreciate a clearer discussion of the settings where LC+ struggles and what might explain these gaps.
- The fine-tuning results are encouraging, but the evidence is not convincing. Table 5 shows that LC (SFT) still falls behind LC+ on SimpleQA by around 0.09 AUROC, along with worse calibration. Interestingly, the picture changes on NQ-Open, where calibration improves substantially, yet the AUROC advantage over SU is less than 0.01. Without uncertainty estimates, it is difficult to judge how meaningful this small difference is. The inconsistency across datasets also leaves some questions about the generalizability of the fine-tuning gains.
Questions for authors
- What explains the weaker LC+ results in Tables 2, 3 and Figure 4 (left)? Is this mainly due to models failing to express meaningful uncertainty, or limitations of the confidence mapper?
- How robust are the fine-tuning gains? Could the authors provide confidence intervals for the small AUROC improvement over SU on NQ-Open? Why does SFT outperform LC+ on NQ-Open but not on SimpleQA?
- To what extent do abstentions contribute to the reported improvements? Could the authors report answer coverage and AUROC excluding abstentions to clarify whether the gains persist among attempted answers?
Thank you for the careful review. We address both weaknesses and all three questions below.
W1. Human alignment and filtering
An expression can admit multiple reasonable interpretations. More independent judgments could support a distribution over perceived confidence. With five ratings per expression, we target a scalar mean of retained judgments, not the full distribution.
This motivates a scalar target but does not independently justify filtering. Confidence-level bounds may exclude legitimate disagreement, and aggregate MSE may obscure errors in sparse categories. We examine these issues separately below.
W2. Interpretation of SFT
SFT combines factual and confidence supervision: gold answers provide content, while SU selects confidence levels. Changes in answers and abstention prevent attributing all improvements to confidence expression.
SFT is a feasibility exploration. The dataset, mapper, and evaluation remain the main contributions, as Reviewer x5S7 also recognized. We will report the accuracy–coverage trade-off explicitly. SFT remains below LC+ on SimpleQA, and its NQ-Open AUROC advantage over SU is only .0079; neither supports universal superiority.
Q1. Human validation, filtering, splits, and per-level errors
We sampled 200 SimpleQA questions for Qwen3-235B-A22B, collecting LC/LC+ outputs and independent human ratings without confidence-level filtering. Both scoring sources use identical responses, correctness labels, and abstention conventions.
Both sources support the same direction, although absolute estimates differ. This supports the sampled comparison, not all models or datasets.
Errors grouped by human mean confidence are:
These actual-output groups differ from the benchmark’s prescribed levels. Errors are larger at the extremes.
Unfiltered targets retain worker-quality checks. Generator-held-out evaluation trains on three generators and tests the fourth, rotating across all four; it is not jointly question- and generator-disjoint.
The original split guarantees expression separation, not verified question or generator isolation. Different test compositions prevent interpreting MSE changes as pure leakage effects. The filtering check also does not evaluate all excluded expressions.
Q2. Accuracy, coverage, uncertainty, and matched control
Saved records matching Table 5 provide:
SFT lowers overall accuracy and coverage while improving answered-only AUROC. Different attempted subsets prevent attributing this solely to confidence supervision.
An answer-only control matches questions, factual content, example counts, and training steps, without SU-conditioned hedging.
On questions both SFT models attempt:
Evidence is stronger on NQ-Open; SimpleQA remains inconclusive. This does not isolate SU alignment from general hedging supervision or hold factual answers fixed.
Q3. SU, perplexity, and ECE
The inspected SU implementation uses largest-cluster frequency, c=max_k(n_k/N), with ten samples by default. The prediction is the first answer in the largest cluster. This is semantic-consistency confidence, not normalized entropy.
DeBERTa MNLI assesses both pairwise directions. Grouping requires no contradiction and at least one entailment.
Perplexity confidence is exp(mean(token log-probabilities)), the inverse corresponding perplexity. Table 5 configurations use 15 equal-width ECE bins, weighted by sample count, skipping empty bins and excluding NOT_ATTEMPTED responses.
We are verifying token selection, abstention handling, Table 4’s training path, and the effect of excluding exact-zero confidence from ECE bins.