acceptodds
Under review as a conference paper at ICLR 2027

Reliability and Informativeness of Verbalized Confidence in Language Models

Abstract

Language models are increasingly asked to state how confident they are, and the numbers they give back are reported as calibration evidence. Calibration presupposes a measurement, however: a quantity that carries information about correctness and that stays put under changes to how it is requested. We test both properties across open-weight instruction-tuned models spanning more than an order of magnitude in scale, on five English benchmarks, under five prompt formulations and seven ways of asking for a confidence value. Verbalized confidence separates correct answers from incorrect ones only slightly better than chance, with a mean AUROC of 0.536 against 0.682 for the first-token probability of the same responses. It is also unstable. Holding the model, the question and the model's own answer fixed and changing only the wording of the request, the two formulations used in prior work barely agree, and a model asked how likely its answer is to be wrong does not give a value consistent with what it reports when asked how likely that answer is to be right. Often no value is recorded at all, and much of that missingness belongs to the extraction rule rather than to the model. None of this resolves with scale: within one family at fixed training recipe, accuracy and first-token informativeness both rise steeply across a sixty-six-fold parameter range while the verbalized report stays near chance. A model trained to verbalize confidence behaves differently under the same procedure, which indicates that our measurement registers a trained signal where one is present. The practical consequences follow. Which model looks best calibrated changes with the wording of the request in roughly a third of our comparisons, and abstention built on this signal recovers about a seventh of the accuracy available from token probability. We recommend that checks of informativeness and reliability accompany any reported verbal calibration number.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
verbalized confidence, uncertainty quantification, calibration, measurement validity, evaluation methodology, selective prediction, language model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_j2AyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors analyze verbalized confidence for black-box LLM confidence calibration. They find prompt sensitivity, poor discriminability, and parsing difficulties. This holds over models of parameter counts 0.5B-32.8B.

Critical strengths

None

Critical weaknesses
  • The findings are not significant or novel.
    • The prompt sensitivity of verbalized confidence has been shown in prior work, e.g. Zhao et al. (2024). The incoherence of verbalized confidence (i.e. that the total verbalized confidence over mutually exclusive claims can exceed 1) has also been shown, e.g. Wang and Stengel-Eskin (2026).
    • The poor calibration/discriminability of verbalized confidence has also been shown, e.g. Tian et al. (2023) and Xiong et al. (2024).
    • The parsing difficulties with verbalized confidence seem unaddressed by prior work, but this is not a meaningful contribution, since it is obvious that one should sanity check the parsing rule being used.

References:

Zhao et al., 2024. Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models.

Wang and Stengel-Eskin, 2026. Calibrating Verbalized Confidence with Self-Generated Distractors.

Tian et al., 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.

Xiong et al., 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.

Questions for authors

None

Reviewer_q4E4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Verbalized confidence scores is one approach to determine the confidence a model has in its generated answer. However, existing works relying on verbalized confidence assume this measurement is calibrated, meaning that it can correctly assess the objective it was designed for and the same conditions result in the same value. The authors question this assumption and performed extensive experiments across different model families, scales, and checkpoints to test it. Through their results, they demonstrate how verbalized confidence is not calibrated and can vary greatly under the same conditions, regardless of the model configuration.

Critical strengths

Extensive experiments: The results covered a wide range of model families and scales. The authors also compared tuned models with the corresponding base model. This provides substantial evidence to show how verbalized confidence is not a reliable score. On a side note, the tables referenced in the main paper should not appear in the appendix.

Critical weaknesses
  1. Prior works: There are many other works that analyze verbalized confidence and how it is largely inaccurate, but I don't see it referenced in the paper. Listed below are just a few that I found. These papers either identify an internal reason for overconfident scores or propose a solution (or both), whereas this paper only provides empirical results to prove the claim that verbalized confidence scores are not calibrated.

  2. I find the recommendations rather weak in that they are mitigations rather than solutions to the issue of unreliable verbalized confidence scores. The experiments also do not find any insights in how to tackle this problem (no clear pattern in which model configurations cause verbalized confidence).

[1] Miao, M. M., & Ungar, L. (2026). Closing the confidence-faithfulness gap in large language models. arXiv preprint arXiv:2603.25052.

[2] Zhao, T., He, Y., Zheng, W., Zhang, Y., & Chen, C. (2026). Wired for overconfidence: A mechanistic perspective on inflated verbalized confidence in llms. arXiv preprint arXiv:2604.01457.

[3] Kumaran, D., Conmy, A., Barbero, F., Osindero, S., Patraucean, V., & Veličković, P. (2026). How do llms compute verbal confidence. arXiv preprint arXiv:2603.17839.

[4] Xu, C., Wen, B., Han, B., Wolfe, R., Wang, L. L., & Howe, B. (2025, July). Do language models mirror human confidence? exploring psychological insights to address overconfidence in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 25655-25672).

Questions for authors
  1. In table 4, ConfTuner only barely beats its base model and even achieves a lower AUROC on MNLI. Then, isn't that an indicator that ConfTuner has the same issue as all other LLMs in that it cannot give reliable confidence score? How would other methods targeting unreliable confidence scores perform compared to the paper's results?
  2. When generating the confidence score, the paper mentioned that the model generates eight tokens. What if you restrict it to only one or two (i.e, force the model to only output a percentage from 0 - 100)? This allows you to see the actual logit values of the confidence scores and maybe give more insight.
Reviewer_E6RFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates the informativeness, cross-prompt reliability, and probabilistic coherence of verbalized confidence in instruction-tuned language models. In the evaluated settings, verbalized confidence provides weak discrimination between correct and incorrect answers, changes with elicitation wording, and offers limited selective-prediction benefits relative to token probabilities. A confidence-tuned checkpoint serves as a positive control. The paper recommends reporting informativeness and reliability alongside calibration.

Critical strengths

Useful controlled design. Holding the model’s answer fixed while changing only the confidence request separates elicitation sensitivity from changes in the answer itself.

Relevant positive control. The ConfTuner comparison demonstrates that the pipeline can detect an improved verbalized signal and appropriately limits the scope of the negative findings.

Critical weaknesses

1 The novelty is limited, and the positioning overstates the gap in prior work, such as paper in ref below. More broadly, confidence-based error detection predates modern LLMs, including Hendrycks and Gimpel (2017). These studies do not duplicate the exact fixed-answer protocol, but they substantially weaken the claim that prior work has skipped the underlying question. The remaining contribution is a narrower diagnostic refinement, and the current analysis does not establish a sufficiently distinct insight beyond known limitations of verbalized confidence.

2 The experiments do not sufficiently separate signal failure from protocol effects or establish the significance of the disagreement. The token baseline comes from a separate generation call, while verbalized confidence uses a short output budget and format-sensitive parsing. The primary comparison therefore needs clearer answer alignment and extraction-robustness checks. The reliability analysis also emphasizes low correlations for highly concentrated scores without sufficiently establishing the absolute magnitude and practical impact of the disagreement. The normalized standard deviation of the difference is algebraically related to the same variances and covariance, rather than independent confirmation. What is missing is not a larger benchmark sweep, but targeted controls that distinguish these explanations and clarify what the proposed diagnostics reveal beyond score concentration.

References for my comments:

Hendrycks, D., and Gimpel, K. (2017). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. ICLR. arXiv:1610.02136. Xiong, M., et al. (2024). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ICLR. arXiv:2306.13063. Yang, D., Tsai, Y.-H. H., and Yamada, M. (2024). On Verbalized Confidence Scores for LLMs. arXiv:2412.14737. Ni, S., Bi, K., Yu, L., and Guo, J. (2024). Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence? arXiv:2408.09773. Zhu, J.-Q., and Griffiths, T. L. (2024). Incoherent Probability Judgments in Large Language Models. arXiv:2401.16646.

Questions for authors

Why is cross-prompt reliability a prerequisite for calibration, rather than a separate property?

Are the two confidence signals evaluated on exactly the same answers and items? Section 3.2 describes a separate single-token call for the token baseline. What happens when its prediction differs from the original answer?

Which figures and results should be treated as final? Figure 1 shows 180 settings and 150 wins for token probability, whereas its caption and the text report 102 settings and 89 wins. Figure 2 stops at 14.8B and shows only verbalized AUROC on the right, despite a caption describing a 32.8B endpoint and both signals. Please reconcile these discrepancies so the main comparisons and scaling claim can be verified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.