acceptodds
Under review as a conference paper at ICLR 2027

Hedging as Confidence: Measuring and Improving Linguistic Uncertainty in LLMs

Abstract

Large language models (LLMs) are increasingly used in high-stakes settings, where overconfident responses can mislead users. Reliable confidence estimation has been shown to enhance trust and task accuracy. Yet existing methods face practical barriers: logits are often hidden, multi-sampling is computationally expensive, and verbalized numerical uncertainty (e.g., giving a 0-100 score) deviates from natural communication. We revisit linguistic confidence (LC), where models express uncertainty through hedging language (e.g., probably, might), offering a lightweight and human-centered alternative. To advance this direction, we 1) release the first diverse, large-scale dataset of hedging expressions with human-annotated confidence scores, and 2) propose a lightweight mapper that converts hedges into confidence scores at near-zero cost. Building on these resources, we 3) conduct the first systematic study of LC under standard uncertainty-estimation criteria (calibration and discriminability) across modern LLMs and QA benchmarks, revealing that while most LLMs underperform in expressing reliable LC, carefully designed prompting achieves competitive calibration and discriminability. Finally, we 4) introduce a fine-tuning framework that further improves LC reliability. Taken together, our work positions linguistic confidence as a scalable, efficient, and human-aligned approach to LLM uncertainty estimation, and calls for deeper exploration of this promising yet underexplored direction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
uncertainty quantification, LLM

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_73hUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs can reliably express uncertainty through natural-language hedging (LC). The authors construct a human-annotated dataset of confidence expressions and develop a lightweight mapper to convert them into numerical confidence scores. Experiments across multiple LLMs and QA benchmarks evaluate calibration and discriminability, showing that explicit hedging prompts (LC+) generally improve over vanilla LC, although performance against existing UQ methods remains mixed. The authors further propose a semantic-uncertainty-guided fine-tuning framework, achieving improvements on SimpleQA and NQ-Open, particularly on the latter, without requiring additional hedging instructions at inference time.

Critical strengths
  1. The human-annotated dataset and lightweight mapper are valuable contributions. Unlike fixed phrase-level mappings, the mapper handles complete answer sentences. Table 1 reports roughly 72% lower MSE than the best direct-prompt baseline, with substantially lower latency. I think this provides a practical and efficient tool for studying linguistic confidence.
  2. The systematic evaluation of linguistic confidence is informative. By examining both calibration and discriminability, the paper offers useful insights into spontaneous versus explicitly elicited uncertainty. The fine-tuning results are also encouraging, improving AUROC by about 0.23 and reducing ECE by 0.16 on NQ-Open. Despite mixed results across settings, these findings are worth further investigation.
Critical weaknesses
  1. The competitiveness of LC+ seems somewhat overstated. Although Tables 2 and Table 3 show improvements over vanilla LC, the results are mixed when compared with existing UQ methods. For instance, even with reasoning enabled, LC+ trails VNC by roughly 0.21 AUROC on GPT-5 (Figure 4, left). This suggests that explicit hedging does not always translate into reliable uncertainty estimation. I would appreciate a clearer discussion of the settings where LC+ struggles and what might explain these gaps.
  2. The fine-tuning results are encouraging, but the evidence is not convincing. Table 5 shows that LC (SFT) still falls behind LC+ on SimpleQA by around 0.09 AUROC, along with worse calibration. Interestingly, the picture changes on NQ-Open, where calibration improves substantially, yet the AUROC advantage over SU is less than 0.01. Without uncertainty estimates, it is difficult to judge how meaningful this small difference is. The inconsistency across datasets also leaves some questions about the generalizability of the fine-tuning gains.
Questions for authors
  1. What explains the weaker LC+ results in Tables 2, 3 and Figure 4 (left)? Is this mainly due to models failing to express meaningful uncertainty, or limitations of the confidence mapper?
  2. How robust are the fine-tuning gains? Could the authors provide confidence intervals for the small AUROC improvement over SU on NQ-Open? Why does SFT outperform LC+ on NQ-Open but not on SimpleQA?
  3. To what extent do abstentions contribute to the reported improvements? Could the authors report answer coverage and AUROC excluding abstentions to clarify whether the gains persist among attempted answers?
Reviewer_YASXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies confidence communicated through hedging. It collects 50,000 human ratings of 10,000 generated expressions, filters these into a 1,622-expression evaluation benchmark, and trains a DistilRoBERTa confidence mapper. It compares vanilla linguistic confidence (LC), explicitly prompted hedging (LC+), and numerical or sampling-based estimates across factual QA benchmarks and ten models. LC+ generally improves over LC. A Qwen3-8B fine-tuning experiment uses semantic uncertainty to choose hedging levels and improves the reported LC metrics, especially on NQ-Open.

Critical strengths
  1. The lightweight mapper makes a useful empirical direction practical. Table 1 reports MSE 50.68 versus 183.23 for the best listed LLM mapper on the human benchmark, with 1.32 seconds to map one SimpleQA evaluation run. The annotation effort and continuous scoring extend the paper's evaluation beyond a small fixed hedge vocabulary.

  2. The LC/LC+ comparison is informative across model families: both reported metrics improve in all model–dataset comparisons in Tables 2–3. The distinction between prompted elicitation and the intended vanilla setting is explicit. Table 5's NQ-Open improvement after SFT also provides a promising initial demonstration of learning this behavior.

Critical weaknesses
  1. The human-alignment evidence depends heavily on benchmark construction. Section 3 keeps ratings within one standard deviation of the mean for each generated confidence level, retaining only 1,622 of 10,000 expressions; just 64 are high-confidence and 11 are completely uncertain. This can remove meaningful disagreement and leaves the ends of the scale weakly tested. Expression-level train/test disjointness does not establish question- or generator-level generalization. Since the mapper underlies every downstream LC result, fresh human validation on actual LC/LC+ outputs and sensitivity to filtering are needed to establish that the reported calibration reflects perceived confidence.

  2. The fine-tuning results do not separate improved uncertainty expression from changed answer content and abstention. Appendix A.1 supplies gold answers to generate targets, and Table 8 changes factual answers as well as hedging. ECE excludes abstentions while AUROC includes them, yet Table 5 lacks accuracy, coverage, or answered-only AUROC. The NQ-Open AUROC advantage over SU is 0.0079 without uncertainty estimates; on SimpleQA, SFT remains worse than LC+ on both metrics. Missing definitions of the SU-to-confidence transformation and ECE binning further limit interpretation. These omissions leave the claimed reliability improvement insufficiently established.

Questions for authors

Clarifications and analyses that could change my assessment:

  1. How do mapper errors and downstream conclusions change with fresh human ratings of actual model outputs, without confidence-level filtering? Are the mapper splits disjoint by question and generator, and what are the errors by confidence level?
  2. Can you report accuracy, abstention coverage, answered-only AUROC, and paired uncertainty estimates for Table 5, together with a matched fine-tuning control that separates answer supervision from confidence-level supervision?
  3. What exact semantic uncertainty statistic and transformation produce the [0,1] confidence used in Table 4 and ECE? Please also specify the perplexity-to-confidence transformation and ECE binning.
2 comments

Thank you for the careful review. We address both weaknesses and all three questions below.

W1. Human alignment and filtering

An expression can admit multiple reasonable interpretations. More independent judgments could support a distribution over perceived confidence. With five ratings per expression, we target a scalar mean of retained judgments, not the full distribution.

This motivates a scalar target but does not independently justify filtering. Confidence-level bounds may exclude legitimate disagreement, and aggregate MSE may obscure errors in sparse categories. We examine these issues separately below.

W2. Interpretation of SFT

SFT combines factual and confidence supervision: gold answers provide content, while SU selects confidence levels. Changes in answers and abstention prevent attributing all improvements to confidence expression.

SFT is a feasibility exploration. The dataset, mapper, and evaluation remain the main contributions, as Reviewer x5S7 also recognized. We will report the accuracy–coverage trade-off explicitly. SFT remains below LC+ on SimpleQA, and its NQ-Open AUROC advantage over SU is only .0079; neither supports universal superiority.

Q1. Human validation, filtering, splits, and per-level errors

We sampled 200 SimpleQA questions for Qwen3-235B-A22B, collecting LC/LC+ outputs and independent human ratings without confidence-level filtering. Both scoring sources use identical responses, correctness labels, and abstention conventions.

ScoresLC ECELC+ ECELC AUROCLC+ AUROC
Human means.3126.2189.5268.6741
Mapper.2984.2057.5183.6906

Both sources support the same direction, although absolute estimates differ. This supports the sampled comparison, not all models or datasets.

Errors grouped by human mean confidence are:

ConfidenceOutputsMSE
[0,20)37109.64
[20,40)6394.27
[40,60)10676.85
[60,80)11969.32
[80,100]7598.16
Overall40084.38

These actual-output groups differ from the benchmark’s prescribed levels. Errors are larger at the extremes.

EvaluationMapper MSEGPT-5 MSE
Original benchmark50.68183.23
Same expressions, unfiltered targets78.36187.54
Question-disjoint67.82180.91
Generator-held-out, macro-average73.49186.28

Unfiltered targets retain worker-quality checks. Generator-held-out evaluation trains on three generators and tests the fourth, rotating across all four; it is not jointly question- and generator-disjoint.

The original split guarantees expression separation, not verified question or generator isolation. Different test compositions prevent interpreting MSE changes as pure leakage effects. The filtering check also does not evaluate all excluded expressions.

Q2. Accuracy, coverage, uncertainty, and matched control

Saved records matching Table 5 provide:

DatasetMethodAccuracyCoverageECE excl.AUROC incl.AUROC answered
SimpleQALC4.75%99.49%.7781.4790.4786
SimpleQASFT3.32%92.29%.7550.5769.5418
NQ-OpenLC34.93%95.18%.4396.5006.4957
NQ-OpenSFT25.96%81.66%.2837.7331.6912

SFT lowers overall accuracy and coverage while improving answered-only AUROC. Different attempted subsets prevent attributing this solely to confidence supervision.

An answer-only control matches questions, factual content, example counts, and training steps, without SU-conditioned hedging.

DatasetControl accuracyCoverageAnswered AUROC
SimpleQA4.48%98.89%.4937
NQ-Open34.13%94.49%.5376

On questions both SFT models attempt:

DatasetControl AUROCSU-guided AUROCDifference, paired 95% CI
SimpleQA.4961.5374+.0413 [−.0128, +.0976]
NQ-Open.5418.6815+.1397 [+.0942, +.1846]

Evidence is stronger on NQ-Open; SimpleQA remains inconclusive. This does not isolate SU alignment from general hedging supervision or hold factual answers fixed.

Q3. SU, perplexity, and ECE

The inspected SU implementation uses largest-cluster frequency, c=max_k(n_k/N), with ten samples by default. The prediction is the first answer in the largest cluster. This is semantic-consistency confidence, not normalized entropy.

DeBERTa MNLI assesses both pairwise directions. Grouping requires no contradiction and at least one entailment.

Perplexity confidence is exp(mean(token log-probabilities)), the inverse corresponding perplexity. Table 5 configurations use 15 equal-width ECE bins, weighted by sample count, skipping empty bins and excluding NOT_ATTEMPTED responses.

We are verifying token selection, abstention handling, Table 4’s training path, and the effect of excluding exact-zero confidence from ECE bins.

I appreciate the new human ratings on SimpleQA. They support the finding that prompted hedging makes expressed confidence more reliable on that test. The ratings cover one model, and the mapper makes its largest errors at the extremes of the confidence scale. I would want a broader human check before relying on it for comparisons across models and datasets.

The answer-only control makes the fine-tuning result on NQ-Open more persuasive. The same comparison is less conclusive on SimpleQA, and fine-tuning lowers accuracy and coverage on both datasets. Changes in the answers themselves also make it hard to tell how much of the gain comes from better confidence expression. I see this as an initial demonstration and retain my original score of 2 (weak rejection).

Reviewer_KnLeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper estimates LLM uncertainty via hedges (linguistic confidence, LC) to avoid logits, costly sampling, and unnatural verbalized numeric confidence (VNC). It builds a crowdsourced hedge dataset, trains a lightweight DistilRoBERTa regressor, evaluates calibration (ECE) and discrimination (AUROC), and proposes LoRA SFT with semantic uncertainty (SU) pseudo-labels to make Qwen3-8B output hedges naturally.

Critical strengths
  1. Important problem and useful evaluation tool: Low-cost uncertainty expression matching human interaction is practically valuable; the mapper cuts single-round evaluation from about 600s to 1.32s at near-zero cost and has lower MSE than an LLM judge (50.68 vs. 183.23).
  2. Cross-model diagnostics reveal a gap between internal signals and surface expression: Evaluations on 10 models compare default LC and explicit LC+, showing that models internally sense uncertainty but often fail to express it by default.
Critical weaknesses
  1. The paper claims the “first systematic study of LC” but omits classic psycholinguistic work on mapping probability words to numbers [1], analyses of epistemic markers under default/explicit prompting [2, 3], and prior linguistic-calibration fine-tuning [4]; the DistilRoBERTa plus pseudo-label LoRA pipeline is also methodologically routine.

  2. The authors rely only on “numbers are unnatural” and do not explain why hedges should be more reliable, while ignoring that VNC significantly outperforms LC and even LC+ in several settings (Table 2; Figure 4 left with reasoning-enabled GPT-5); they also do not explain performance drops after LC+ or whether reasoning gains come from accuracy or hedge distribution.

  3. Filtering annotations outside mean ±1σ discards about 75% of annotations and 84% of sentences, causing severe class imbalance and artificially lowering Table 1 mapping error; the mapper’s training and test data also come from the same 200 SimpleQA questions, risking question-level semantic leakage.

  4. Rough SFT design and missing key evidence such as post-fine-tuning QA accuracy (Sections 6.1–6.3; Table 5): SFT uses only 200 questions to generate 8,000 highly repetitive QA pairs (40 paraphrases each), and it is unclear whether target sentences are based on gold answers or model predictions; Table 5 omits post-SFT QA accuracy and refusal rate, and the NQ-Open headers are reversed with LC (SFT) ECE/AUROC values apparently swapped.

References: [1] Wallsten et al. Measuring the vague meanings of probability terms. JEP: General, 1986.

[2] Zhou, Jurafsky, and Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. EMNLP, 2023.

[3] Zhou, Hwang, Ren, and Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. ACL, 2024.

[4] Band, Li, Ma, and Hashimoto. Linguistic calibration of long-form generations. ICML, 2024.

Questions for authors
  1. Positioning versus prior work and mechanistic explanation of LC vs. VNC: Given prior work on hedge calibration and fine-tuning [2–4], what is the core methodological novelty, and why does VNC still significantly outperform LC/LC+ in Figure 4 left and several Table 2 settings?

  2. Mapper performance without ±1σ truncation and with question-level isolation: What is the mapper MSE on all valid annotations without ±1σ filtering, and do the training set and the 1,622 test examples have zero overlap at the question-ID level?

  3. SFT QA accuracy, target-sentence generation, and Table 5 correction: What are Qwen3-8B’s QA accuracy and refusal rate before/after SFT, are SFT targets based on gold answers or sampled model answers, and are the NQ-Open LC (SFT) values in Table 5 indeed swapped?

2 comments

Thank you for the detailed review. We address each weakness and question below.

W1. Positioning and methodological novelty

We agree that linguistic confidence has substantial prior literature. We will discuss Wallsten et al., Zhou et al. (2023, 2024), and Band et al. (2024), and replace the broad “first systematic study” wording with a precise statement of scope.

Our main contributions are the human-annotated resource, inexpensive continuous mapper, and evaluation across ten LLMs and three QA benchmarks. DistilRoBERTa and LoRA are established components; we do not claim architectural novelty.

W2. Motivation and comparison with VNC

Naturalness motivates studying LC but does not imply superiority over VNC. Answers communicate confidence even without a request for probabilities; we investigate whether this confidence tracks correctness.

We agree that LC+’s competitiveness was overstated. Its relative performance depends on the model and metric. Table 2 shows lower ECE for LC+ than VNC on all three benchmarks for Qwen3-235B-A22B, but higher AUROC for VNC on all three for GPT-OSS-120B.

Tables 2–3 show improvements from LC to LC+ in both metrics, rather than deterioration after prompting. However, reasoning changes both correctness and confidence expression, so Figure 4 does not isolate their contributions. We will qualify these interpretations.

W3. Filtering and potential leakage

Section 3 retains 12,762 of 50,000 ratings after confidence-level filtering, then requires at least three retained ratings per expression, yielding 1,622 expressions. Sentence retention therefore also depends on the minimum-rating requirement.

Five ratings per expression limit estimation of a full interpretation distribution, motivating a scalar mean. This does not independently justify filtering, which may remove legitimate disagreement. We acknowledge the resulting selection effects and limited representation of confidence extremes.

The original split guarantees disjoint expressions, not verified zero question-ID overlap. Question-level generalization therefore requires a separate evaluation, discussed under Q2.

W4. SFT design and missing measurements

SFT is an initial feasibility test. The dataset, mapper, and evaluation remain the main contributions, as Reviewer x5S7 also recognized.

The 8,000 pairs contain 200 questions with 40 alternative responses each, not 8,000 independent factual examples. Targets combine gold-answer content with SU-selected confidence levels, so the training can change both answers and confidence expression. We will not attribute all improvements to confidence supervision.

Q1. Core contribution and why VNC can outperform LC

Our contribution is a practical measurement resource and comparative empirical study, rather than the invention of linguistic calibration. VNC outperforming LC in some settings is an important finding, not inconsistent with this objective.

For reasoning-enabled GPT-5:

MethodAccuracyECEAUROC
LC49.94%.2780.5565
LC+49.32%.2601.6204
VNC49.65%.1979.8286

The LC+–VNC AUROC gap occurs despite similar aggregate accuracy. Possible explanations include insufficiently differentiated hedges and mapper limitations; the current comparison does not isolate them. We will describe LC+ as an elicitation probe, not a demonstrated upper bound.

Q2. Unfiltered MSE and question-level isolation

We evaluated the same 1,622 held-out expressions using targets averaged before confidence-level filtering but after worker-quality checks. Separately, we retrained with question-disjoint partitions.

SettingMapper MSEGPT-5 mapper MSE
Original filtered benchmark50.68183.23
Same expressions, unfiltered targets78.36187.54
Question-disjoint evaluation67.82180.91

MSE uses the 0–100 scale. Absolute error increases, while the mapper retains its relative advantage.

The additional split has zero question overlap; this does not establish zero overlap in the original split. Different test compositions prevent attributing the entire MSE change to overlap. The fixed-expression analysis also does not establish performance on excluded expressions.

Q3. SFT accuracy, refusals, targets, and Table 5

Targets use gold answers for factual content and base-model SU for confidence levels. Sampled base-model answers are not the factual targets. The completely uncertain level requests refusals. Training questions are excluded from evaluation.

Saved records matching Table 5 provide:

DatasetMethodAccuracyCoverageECE excl.AUROC incl.AUROC answered
SimpleQALC4.75%99.49%.7781.4790.4786
SimpleQASFT3.32%92.29%.7550.5769.5418
NQ-OpenLC34.93%95.18%.4396.5006.4957
NQ-OpenSFT25.96%81.66%.2837.7331.6912

Accuracy uses all questions; coverage is the attempted fraction. Abstention rates are 0.51% versus 7.71% on SimpleQA and 4.82% versus 18.34% on NQ-Open.

SFT improves uncertainty metrics while reducing accuracy and coverage. Answered-only AUROC improves, but differing attempted subsets prevent a causal interpretation. We will make this trade-off explicit.

Table 5 is not swapped: NQ-Open SFT ECE is .2837 and AUROC is .7331. Its AUROC advantage over SU is only .0079, while SFT remains below LC+ on SimpleQA. We will qualify these comparisons.

Reviewer_x5S7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LLMs deployed in the real-world with people should be able to handle -- and express -- their uncertainty. This uncertainty need not be expressed numerically, but in language. The authors here collate a large-scale dataset of linguistic expressions of uncertainty from human-data and train a mapper from natural language to the level of uncertainty. They then evaluate a set of LLMs against this dataset and propose a new fine-tuning method for potentially improving the expression of LLMs' uncertainty.

Critical strengths

I think the authors do an excellent job of taking on an important problem and approaching it from multiple angles (data collection; interesting operations over the data [confidence classifier]; evaluation against LLMs; productive method to potentially improve gaps found in eval).

I think the work can have a substantive contributions and other researchers could benefit from the dataset in particular that the authors collect.

Critical weaknesses

I found the presentation of the results somewhat confusing and unclear how robust they are, as no error bars / CIs are reported. How variable are the results? Are the differences in performance measures actually significant? This applies both to the evaluation, and the authors' own fine-tuning work (are the changes meaningfully different?)

I think the authors' fine-tuning extension is probably the weakest contribution (it wasn't clear to me how much value it adds, as the authors note). I think the paper with its dataset and eval alone is already quite strong and a nice contribution!

I also think the authors could benefit from a bit more substantive / broad discussion of related work (see questions).

Questions for authors
  • Did the authors attain ethics approval for their human data collection (I couldn't see this in the overview?)
  • Did any MTurk participants engage in the study more than once (I see they were allowed to 5 times?)
  • Less a question, but more a comment --- there is a broader literature on linguistic expressions of uncertainty the authors might benefit from connecting to, e.g., https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(22)00060-2 (more on differences in linguistic vs numeric uncertainty -- and see lots of their related work; https://arxiv.org/abs/2305.01020 -- on LLMs mapping from ambigious linguistic expressions to numerical uncertainty; https://arxiv.org/html/2302.13439v2 -- on other LLM linguistic uncertainty w/ crowdsourcing)

Comments on the paper

1 comment
Area_Chair_cLE6OpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.