acceptodds
Under review as a conference paper at ICLR 2027

Calibrating Verbalized Confidence with Rubrics

Abstract

Large language models (LLMs) can assess the correctness of their own responses through verbalized confidence, helping users decide whether to adopt the response or seek further review. However, models often exhibit overconfidence when their responses are incorrect, limiting the practical value of these estimates. Existing research addresses this problem by refining how confidence judgments are elicited and aggregated, yet whether expressed confidence is reliable under more stringent and explicit verification criteria remains underexplored. In this paper, we systematically analyze models' verbalized confidence through progressively stricter rubric-based checks and propose two confidence calibration methods. The proposed methods integrate verification signals from these checks with the model's holistic self-assessment, thereby mitigating overconfidence. Experiments on eight benchmarks spanning diverse reasoning tasks and domain knowledge, across three model families and three model scales ranging from B to B, demonstrate improved calibration while preserving discrimination between correct and incorrect responses. Further analysis shows that rubric-based verification provides useful signals beyond direct confidence, and that the gains cannot be attributed merely to more conservative confidence estimates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM uncertainty, confidence calibration, verbalized confidence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hbe6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study LLM confidence estimation, specifically verbalized confidence.

They propose Rubric Calibration (RC) which combines a direct model-output confidence estimate with five rubric scores from an additional prompt. A second proposed variant RC-A then also applies the rubric-based correction to a subset of responses.

The propposed confidence scores are compared with various baselines across 8 benchmarks/datasets totalling 3,910 questions, using three different LLMs (Qwen3-8B, Gemma-3-12B-IT, Llama-3.1-8B-Instruct).

Critical strengths
  • The paper is well written and easy to follow overall, the studied problem is quite interesting and important, and the proposed method is simple and seems to perform quite well.
Critical weaknesses
  • The proposed CRCC_{\mathrm{RC}} in equation (2) is rather heuristic, as there is no clear justification for why DmeanD_{\mathrm{mean}} and RmeanR_{\mathrm{mean}} should be multiplied together (instead of e.g. averaged).

  • The proposed extension CRC−AC_{\mathrm{RC-A}} in equation (4) is also heuristic (no clear justification for why the gate should be based specifically on high values of ARA_{\mathrm{R}} or CRCC_{\mathrm{RC}}) and it requires the thresholds λa,λb\lambda_a, \lambda_b to be set. Moreover, RC-A seems to have been proposed based on empirical observations on exactly the same 3,910 benchmark questions which later are used in the main evaluation, RC and RC-A are thus never compared on truly external data.

Questions for authors

Questions/suggestions:

  • Why are DmeanD_{\mathrm{mean}} and RmeanR_{\mathrm{mean}} multiplied together? Why not e.g. average them? Is there some kind of probabilistic/mathematical justification for this?

  • "RC-A seems to have been proposed based on empirical observations on exactly the same 3,910 benchmark questions which later are used in the main evaluation, RC and RC-A are thus never compared on truly external data", do you agree with this from above? The evaluation includes two additional LLMs (which demonstrates some form of generalization), but no external datastes/benchmarks?

  • What are the main limitations of your approach? I think this should be at least briefly discussed/mentioned somewhere in the main paper.

  • Based on Appendix C.4 Inference Budget: I think this should be more explicitly evaluated, using fixed-budget baselines. For example, extend Table 2 with versions of "Direct Mean" and "Rubric Mean" that uses V = 8 calls (instead of V = 4), to match the total call budget of RC and RC-A?

  • It's not quite clear to me how the "Paraphrase Mean" baseline method in Table 1 differs from "Direct Mean" in Table 2 (the results are similar but not identical)?

  • Why specifically K=5 verification levels? Based on Figure 11 in the appendix, it seems likely that increasing this beyond K=5 would further improve the overall results?

  • How did you come up with the "Rubric scoring (origin)" prompt in the appendix? Is this based on some previous paper? Why specifically these "Surface validity", "Local correctness", "Logical sufficiency", "Adversarial robustness", "Full verification" as the five levels? And why e.g. "The stated reasoning is sufficient to support the conclusion, without decisive gaps or unjustified leaps" as the prompt for "Logical sufficiency"? At least to me, this all seems a bit arbitrary?

  • Appendix A.5, "For each elicitation type, three additional English prompt paraphrases are generated by an LLM": Could you perhaps give these additional prompts as well? And, which LLM is used to generate these?

Minor things:

  • Figure 3: What's v1v_1 to vnv_n here? Why the same "v" letter for both direct confidence and rubric profile? And, should it be d1,...,dVd_1, ..., d_V instead of d1,...,dnd_1, ..., d_n?
  • In Section 4.1, PR-P and PR-N are not explained/defined at all?
  • Section 4.2: I would probably mark/format the "ARA_R only" and "CRCC_{RC} only" in some way in the text here, to avoid confusion.


Clarification of rating:

I quite like this paper overall, I found it quite interesting. But, I think there are a number of clarifications required before it potentially could be accepted.

10 comments

Dear Reviewer hbe6,

We hope this finds you well. As the discussion period draws to a close, we would like to respectfully confirm whether we have sufficiently addressed your concerns.

If any questions remain, we would be happy to provide further clarification. Thank you again for your time and constructive comments.

Best regards, The authors

Thank you for the response.

I have read the other reviews and all author responses.

The other reviews are quite similar to mine, I don't think they give me any clear reason to increase my score.

The authors provided a quite solid and comprehensive rebuttal. I think they responded quite well overall. My concerns have partly been adressed, and I would now rate the paper a bit higher than before.

But, I remain quite borderline on the paper. I still think the proposed method seems rather heuristic, and I'm not sure how much impact this would have in practice.

My rating of the paper is higher now than before the rebuttal, but I'm reluctant to increase my score all the way to "3: Weak acceptance". I'm borderline on the paper, leaning slightly towards accept.

Thank you for reading all the reviews and responses and for carefully reconsidering our paper. We genuinely appreciate your recognition of our rebuttal and your explanation of the concerns that remain.

RC is based on an explicit correction principle, which uses rubric support as a response-specific factor to adjust direct confidence: weaker verification support leads to a larger correction. Although RC-A incorporates heuristic design choices, extensive experiments across eight benchmarks and three model families support its effectiveness under a shared default configuration. Table 6 further shows consistent model-averaged gains across neighboring configurations.

Recent studies have demonstrated the practical value of heuristic confidence-guided strategies. Rule-Based Confidence Reasoning [1] uses predefined confidence thresholds for answer selection, while Kumaran et al. [2] show that threshold-based confidence policies govern abstention decisions, with verbalized confidence independently predicting such behavior. DINCO [3] improves verbalized confidence calibration through simple normalization and consistency-based aggregation. These findings suggest that heuristic confidence-based strategies can offer meaningful benefits when empirically validated. Moreover, improving verbalized confidence calibration can support more reliable uncertainty communication and confidence-guided decisions [2, 3], highlighting its potential relevance to both human–AI interaction and model decision-making.

We would be grateful if you could share which specific aspect remains most concerning. If a focused clarification or additional analysis would help strengthen your confidence in the work, we would be very happy to address it. Thank you again for your time and thoughtful consideration.

References

[1] Huang et al. To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts. ICLR, 2025.

[2] Kumaran et al. Causal evidence that language models use confidence to drive behaviour. Nature Machine Intelligence, 2026.

[3] Wang and Stengel-Eskin. Calibrating Verbalized Confidence with Self-Generated Distractors. ICLR, 2026.

Reviewer_FTPqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies overconfidence in verbalized confidence and proposes a rubric-based calibration method. In addition to asking the model for a single overall confidence, the authors ask the same model to score the candidate solution under five increasingly strict verification levels, and multiply the mean direct confidence by the mean rubric score (RC). An adaptive variant (RC-A) applies the rubric factor once more when the rubric profile is highly repeatable across prompt paraphrases or when the RC confidence is still high. The method needs no training and no access to model internals. It is evaluated on various reasoning and knowledge benchmarks across several model families and sizes, together with controls that check whether the rubric scores carry more than a uniform reduction of confidence.

Critical strengths
  • The paper addresses a timely and practically relevant problem, and the method is simple and applicable to black-box models without any training.
  • The evaluation is fairly broad, and the writing is easy to follow.
Critical weaknesses
  • One of my main concerns is where the reported calibration gain actually comes from. To my understanding, RC alone is not clearly better than the strongest baselines on calibration (it is behind SteerConf on Qwen3-8B and roughly on par with Self-Probing on Gemma-3-12B), and the gains in Table 1 come from RC-A, which simply applies the rubric factor a second time. However, Table 6 in the appendix shows that applying the second factor to every response without any gate gives results very close to RC-A, so the adaptive gate itself seems to add little. In addition, the matched scaling and shuffled pairing controls in Section 5.1 are only reported for RC, not for RC-A. It would be helpful if the authors could show that the gain of RC-A over RC is more than another uniform reduction of confidence. Please correct me if I misunderstood the setup.
  • I am also not sure whether the comparison with the baselines uses a comparable inference budget. RC and RC-A use eight evaluator calls per response, whereas the strongest baselines use five and Self-Probing uses two. Appendix C.2 reduces the number of paraphrases, but there is no comparison where the baselines get the same number of calls. Related to this, the AUROC of RC-A is slightly below SteerConf on two of the three models, so the main benefit is on ECE and Brier, which are the metrics most directly affected by lowering confidence.
  • The two thresholds of RC-A are fixed at 0.9, but it is unclear how this value was chosen. The paper introduces 0.9 as an illustrative threshold in Section 3.3 and then uses it as the default in all experiments. If I understood correctly, the motivating analysis in Section 3 uses the same 3,910 responses that are evaluated in Table 1, and the sensitivity analysis in Appendix C.1 also runs on the same benchmarks. The authors also note that Llama-3.1-8B is quite sensitive to the threshold. Choosing the thresholds on held-out data would make the results more convincing.
  • Some details needed for reproduction seem to be missing. It would be helpful to state which LLM generated the prompt paraphrases and to provide the paraphrased prompts themselves, since the method depends on them. I also could not find how the correctness labels were determined for each benchmark (e.g., exact match or an LLM judge). Please let me know if I have overlooked these details.
Questions for authors
  • Please refer to the above weaknesses.
2 comments

Thank you for the detailed response and the additional experiments. The authors addressed most of my concerns, so I raise my score. I hope the discussed points will be well reflected in the final manuscript.

Thank you for your positive assessment and for carefully reviewing our responses and additional experiments. We greatly appreciate your constructive feedback and have updated the manuscript to reflect the clarifications discussed in our responses.

Reviewer_xgC5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether specified verification criteria can calibrate LLM's verbalized confidence. Its proposed method to collect direct confidence and rubrics scores from surface validility to full verification. Rubric calibration(RC) multiples average direct confidence with average rubric scores. RC-A used rubric profile repeatability and current confidence to do selectively calibration. The authors report better calibration results on 8 benchmarks and multiple model families and model scales. More importantly, matched scaling and shuffled pairing show that rubrics scores values more than confidence decreasing.

Critical strengths
  1. This paper gives clear motivation and effective method.
  2. The controls for generic confidence reduction are strong. Matched scaling reduces direct confidence by a constant chosen to match RC's average confidence. Sufffled pairing exchanges rubrics profiles between responses within the same benchamrk and with same direct-confidence mean. These results support the claim that rubric scores carry information about individual answers, beyond their overall tendency to be lower.
  3. This paper conducts extensive experiments, Table1 shows improved ECE, Brier score. Tables2 analyzes two category rubrics scores and gate conditions.
Critical weaknesses
  1. This study does not give enough support that five rubric levels are necessary. The strictest level already asks whether the entire solution is fully correct. It is therefore an important simpler alternative to the complete rubric.
  2. The interpretation and stability of the repeatability gate need stronger support. The centered cosine similarity measures agreement in the shape of rubric profiles. Because every rubric prompt imposes the same increasingly strict ladder, similar descending profiles may arise partly from that shared structure.
  3. The method assumes that downward correction is appropriate. They can only decrease confidence. This makes them plausible remedies for overconfidence, but it limits their applicability to already calibrated or underconfident estimators.
  4. Whether adaptive rule generalized to ood datasets is unclear.
Questions for authors
  1. How were method development and evaluation separated? Can you evaluate the frozen method on unseen benchmarks or report a protocol that holds out entire benchmarks during development?

  2. Does the complete rubric outperform strictest-level alternatives? How do r_5, Dr_5, and a standalone full-verification prompt compare with RC and RC-A? Do the easier levels improve performance beyond what level 5 already provides?

  3. What independent information does profile repeatability contribute? Does A_R remain predictive after controlling for rubric mean, level-5 confidence, profile variance, and the level-1-to-level-5 drop within each benchmark? How common are flat or nearly flat profiles, and how sensitive are results to score rounding?

1 comment

Dear Reviewer xgC5,

We hope this finds you well. As the discussion period draws to a close, we would like to respectfully confirm whether we have sufficiently addressed your concerns.

If any questions remain, we would be happy to provide further clarification. Thank you again for your time and constructive comments.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.