acceptodds
Withdrawn from ICLR 2027

Retrieval-Augmented Linguistic Calibration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Linguistic cues such as "I believe" and "probably" offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic confidence expressions remains underexplored. In particular, co-occurring linguistic cues, contextual variation, and subjective audience interpretation pose unique challenges. We therefore model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing interpretation variability that scalar representations discard. Within this distributional framework, we introduce faithfulness as a complementary evaluation dimension and present Faithfulness Divergence (FD), an information-theoretic metric quantifying the surprise induced in audience beliefs upon truth revelation. Building on these foundations, we present Retrieval-Augmented Linguistic Calibration (RALC), a lightweight post-hoc pipeline that propagates calibrated confidence signals back into natural language via retrieval-augmented rewriting. Across three QA benchmarks and five LLM families, RALC improves in-domain faithfulness and calibration up to 66% and 58%, respectively, outperforming black-box and grey-box calibration baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
calibration, LLMs, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MsUFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the calibration of confidence expressed through natural-language hedging. It represents perceived confidence as a Beta distribution, introduces Faithfulness Divergence (FD) as a complementary diagnostic, and proposes Retrieval-Augmented Linguistic Calibration (RALC). The pipeline calibrates the distribution mean using Platt scaling, retrieves hedging expressions with similar confidence profiles, and rewrites the original response. Experiments across five model families and three QA benchmarks report improvements in linguistic-space calibration and faithfulness, including comparisons with prompted hedging and direct Beta-guided rewriting.

Critical strengths

The paper addresses a useful gap between numerical calibration and communicating confidence to users. RALC provides an explicit mechanism for converting calibrated numerical signals into natural-language responses. Its compatibility with linguistic confidence, token probabilities, and semantic self-consistency makes the pipeline relevant beyond a single confidence estimator.

The direct Beta-guided rewriting comparison provides meaningful evidence for lexical retrieval. Figure 3 reports lower generalized ECE for RALC than for direct Beta-guided rewriting, 0.137 versus 0.164, together with higher signal-to-language rank correlation, 0.953 versus 0.776, and similar content-preservation scores. This supports the practical value of grounding rewriting in retrieved expressions.

Critical weaknesses

The interpretation of distributional spread as human disagreement is not sufficiently established. The same LLM ensemble constructs lexical confidence profiles and evaluates rewritten responses (Section 5.1 and Appendix D.1). Appendix C.2.2 provides useful correlations with human confidence annotations, but these primarily establish agreement in confidence levels rather than validating the estimated dispersion or Beta concentration for contextualized responses. Because concentration motivates both distributional retrieval and FD, this leaves an important part of the central contribution under-validated. Appendix B.3 appropriately describes FD as a diagnostic rather than a proper scoring rule; the concern is its empirical interpretation, not the absence of properness.

The incremental benefit of distributional retrieval over mean-only calibration and retrieval remains unclear. Section 4.1 calibrates only the mean while preserving concentration. Direct Beta-guided rewriting removes retrieval altogether, so it does not isolate the value of matching distributional shape. A matched mean-only retrieval comparison is needed to establish whether the improvements depend on the distributional framework or primarily on calibrated mean confidence and suitable hedging expressions.

Questions for authors

Can the existing human annotations support an assessment of agreement in dispersion, beyond correlations between confidence levels? Please clarify whether the evaluated items overlap with the human reference profiles supplied in the evaluator prompt. If independent validation is unavailable, could the claims about human disagreement be narrowed accordingly?

With the calibrator, lexicon, retrieval depth, and rewriting model fixed, how does retrieval based only on mean confidence compare with Wasserstein-based retrieval? This would directly clarify the contribution of distributional matching.

Are correctness labels recomputed after rewriting, and how are NOT_ATTEMPTED responses treated? Reporting answer-change and correctness-change rates would help interpret calibration improvements alongside the entailment-based content-preservation score.

Reviewer_Y3ASOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Retrieval-Augmented Linguistic Calibration (RALC), a post-hoc framework for calibrating confidence expressed in natural language. The authors model linguistic confidence as a Beta distribution over perceived probabilities of correctness, capturing both average confidence and variability in reader interpretations. They introduce Faithfulness Divergence (FD), an instance-level metric based on concentration-weighted Bayesian surprise, and a retrieval-augmented pipeline that calibrates distribution means using Platt scaling and rewrites responses with appropriate hedging expressions. Experiments across five LLM families and three QA benchmarks report improvements in generalized ECE and FD, including in cross-domain settings.

Critical strengths
  • Connecting confidence calibration to natural-language generation. The paper addresses the practical problem of translating calibrated confidence estimates into natural-language expressions rather than providing numerical confidence scores alone. The proposed retrieval-based rewriting framework applies to multiple upstream confidence signals and requires no changes to the target model.

  • Broad experimental evaluation. The authors evaluate the framework across five LLM families, three QA benchmarks, and three confidence signals, including cross-domain calibration settings.

Critical weaknesses
  1. Faithfulness Divergence (FD) treats variability in readers' interpretations of linguistic confidence as a Bayesian prior over correctness, using concentration as the effective sample size. However, these represent different sources of uncertainty, and this interpretation is not sufficiently justified. Moreover, the evaluation primarily shows that FD satisfies the authors' predefined notion of surprise, rather than establishing its validity as a measure of faithful communication of confidence. For a fixed mean and correctness outcome, more dispersed distributions can receive lower FD scores, potentially favoring expressions that readers interpret less consistently. This concern extends to alternative confidence signals (Section 4.3), where concentration reflects different quantities rather than reader agreement.

  2. Although the paper motivates linguistic confidence as a more intuitive interface, modeling variability in reader interpretations does not necessarily resolve ambiguity in communication. RALC calibrates only the distribution mean while preserving concentration, without validating whether the estimated reader agreement is accurate or maintained after rewriting. The evaluation also relies primarily on an LLM ensemble rather than direct human assessment. Furthermore, the experiments lack a scalar-only calibration baseline that uses the calibrated confidence score to guide linguistic rewriting. Comparing this baseline against RALC would help establish whether distributional modeling provides meaningful benefits beyond standard scalar calibration.

Questions for authors
  1. Faithfulness Divergence (Sections 3.2 and 5.2): Why should agreement among readers be treated as the strength of a Bayesian prior over correctness? Beyond the expected behavior shown in the experiments, what evidence supports FD as a measure of faithful confidence communication?

  2. Distributional vs. scalar calibration (Section 5.3): Since RALC only calibrates the distribution mean, could the authors compare against a scalar-only baseline that uses the calibrated confidence score to guide rewriting? This would help clarify the benefit of distributional modeling.

  3. Reader agreement (Section 4.1): How well does the LLM ensemble capture actual disagreement among human readers? Does the rewritten response preserve the intended concentration?

  4. Contextual interactions (Section 4.2): The paper emphasizes interactions between multiple linguistic cues, but the lexicon is built using individual expressions. How does the method account for these interactions in the final responses?

Reviewer_crjAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper models statement-level linguistic confidence as a Beta distribution to capture both central perceived probability and reader agreement concentration. It introduces Faithfulness Divergence (FD) to quantify instance-level Bayesian surprise upon truth revelation and proposes Retrieval-Augmented Linguistic Calibration (RALC), a post-hoc pipeline that applies Platt scaling to distribution means and rewrites responses using hedging expressions retrieved via 1-Wasserstein distance. Experiments evaluate RALC across five LLMs and three QA benchmarks using linguistic confidence, token probability, and semantic uncertainty signals.

Critical strengths
  1. Distributional formulation: Modeling linguistic confidence as a Beta distribution captures reader interpretation variance beyond scalar scores.

  2. Signal-agnostic pipeline: RALC operates post-hoc in linguistic space without modifying model weights and supports multiple upstream uncertainty signals.

  3. Broad empirical evaluation: The framework is tested across five open-source model families and three diverse QA benchmarks with full prompt templates provided.

Critical weaknesses
  1. Flawed concentration preservation: Under semantic uncertainty, concentration equals the fixed sample count (N=20N=20), making it invalid to preserve and match to linguistic agreement via 1-Wasserstein distance.

  2. Method-of-moments discontinuity: Setting concentration to TZT_Z at zero variance while dividing by vZv_Z for vZ>0v_Z > 0 causes concentration to explode near zero, inflating uncalibrated Token Probability FD to 68.8868.88.

  3. Circular evaluation and data leakage: The same models build the lexicon and evaluate rewritten outputs without human studies, while the evaluator prompt leaks in-context references from the validation benchmark.

  4. Missing discrimination metrics and mean collapse: The paper omits AUROC and compresses calibrated means into [0.5,0.95][0.5, 0.95] near the dataset base rate, reducing ECE via global shifting rather than instance-level discrimination.

  5. Negligible retrieval gain: Removing lexicon retrieval (Direct Beta-Guided Rewrite) yields better entailment (0.9600.960 vs. 0.9560.956) and nearly identical FD (0.7440.744 vs. 0.7260.726), showing gains come from Platt scaling.

Questions for authors
  1. Why is the sampling hyperparameter N=αsem+βsem=20N = \alpha^{\text{sem}} + \beta^{\text{sem}} = 20 treated as target reader agreement concentration during 1-Wasserstein lexicon retrieval?

  2. What is the pre- and post-calibration AUROC in the linguistic space, and why do calibrated confidence means almost never fall below 0.50.5?

  3. How does the method-of-moments discontinuity as vZ→0+v_Z \to 0^+ affect FD, and why does lexicon fitting use MLE while evaluation uses MoM?

  4. How are NOT_ATTEMPTED grades mapped to binary y∈{0,1}y \in \{0, 1\}, and what caused the ECE spike to 0.9250.925 on SQuAD 2.0 (Qwen3-8B) and FD degradation on MMLU (GPT-OSS-20B)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.