- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors analyze verbalized confidence for black-box LLM confidence calibration. They find prompt sensitivity, poor discriminability, and parsing difficulties. This holds over models of parameter counts 0.5B-32.8B.
Critical strengths
None
Critical weaknesses
- The findings are not significant or novel.
- The prompt sensitivity of verbalized confidence has been shown in prior work, e.g. Zhao et al. (2024). The incoherence of verbalized confidence (i.e. that the total verbalized confidence over mutually exclusive claims can exceed 1) has also been shown, e.g. Wang and Stengel-Eskin (2026).
- The poor calibration/discriminability of verbalized confidence has also been shown, e.g. Tian et al. (2023) and Xiong et al. (2024).
- The parsing difficulties with verbalized confidence seem unaddressed by prior work, but this is not a meaningful contribution, since it is obvious that one should sanity check the parsing rule being used.
References:
Zhao et al., 2024. Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models.
Wang and Stengel-Eskin, 2026. Calibrating Verbalized Confidence with Self-Generated Distractors.
Tian et al., 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.
Xiong et al., 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.
Questions for authors
None