- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the calibration of confidence expressed through natural-language hedging. It represents perceived confidence as a Beta distribution, introduces Faithfulness Divergence (FD) as a complementary diagnostic, and proposes Retrieval-Augmented Linguistic Calibration (RALC). The pipeline calibrates the distribution mean using Platt scaling, retrieves hedging expressions with similar confidence profiles, and rewrites the original response. Experiments across five model families and three QA benchmarks report improvements in linguistic-space calibration and faithfulness, including comparisons with prompted hedging and direct Beta-guided rewriting.
Critical strengths
The paper addresses a useful gap between numerical calibration and communicating confidence to users. RALC provides an explicit mechanism for converting calibrated numerical signals into natural-language responses. Its compatibility with linguistic confidence, token probabilities, and semantic self-consistency makes the pipeline relevant beyond a single confidence estimator.
The direct Beta-guided rewriting comparison provides meaningful evidence for lexical retrieval. Figure 3 reports lower generalized ECE for RALC than for direct Beta-guided rewriting, 0.137 versus 0.164, together with higher signal-to-language rank correlation, 0.953 versus 0.776, and similar content-preservation scores. This supports the practical value of grounding rewriting in retrieved expressions.
Critical weaknesses
The interpretation of distributional spread as human disagreement is not sufficiently established. The same LLM ensemble constructs lexical confidence profiles and evaluates rewritten responses (Section 5.1 and Appendix D.1). Appendix C.2.2 provides useful correlations with human confidence annotations, but these primarily establish agreement in confidence levels rather than validating the estimated dispersion or Beta concentration for contextualized responses. Because concentration motivates both distributional retrieval and FD, this leaves an important part of the central contribution under-validated. Appendix B.3 appropriately describes FD as a diagnostic rather than a proper scoring rule; the concern is its empirical interpretation, not the absence of properness.
The incremental benefit of distributional retrieval over mean-only calibration and retrieval remains unclear. Section 4.1 calibrates only the mean while preserving concentration. Direct Beta-guided rewriting removes retrieval altogether, so it does not isolate the value of matching distributional shape. A matched mean-only retrieval comparison is needed to establish whether the improvements depend on the distributional framework or primarily on calibrated mean confidence and suitable hedging expressions.
Questions for authors
Can the existing human annotations support an assessment of agreement in dispersion, beyond correlations between confidence levels? Please clarify whether the evaluated items overlap with the human reference profiles supplied in the evaluator prompt. If independent validation is unavailable, could the claims about human disagreement be narrowed accordingly?
With the calibrator, lexicon, retrieval depth, and rewriting model fixed, how does retrieval based only on mean confidence compare with Wasserstein-based retrieval? This would directly clarify the contribution of distributional matching.
Are correctness labels recomputed after rewriting, and how are NOT_ATTEMPTED responses treated? Reporting answer-change and correctness-change rates would help interpret calibration improvements alongside the entailment-based content-preservation score.