- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors study LLM confidence estimation, specifically verbalized confidence.
They propose Rubric Calibration (RC) which combines a direct model-output confidence estimate with five rubric scores from an additional prompt. A second proposed variant RC-A then also applies the rubric-based correction to a subset of responses.
The propposed confidence scores are compared with various baselines across 8 benchmarks/datasets totalling 3,910 questions, using three different LLMs (Qwen3-8B, Gemma-3-12B-IT, Llama-3.1-8B-Instruct).
Critical strengths
- The paper is well written and easy to follow overall, the studied problem is quite interesting and important, and the proposed method is simple and seems to perform quite well.
Critical weaknesses
-
The proposed in equation (2) is rather heuristic, as there is no clear justification for why and should be multiplied together (instead of e.g. averaged).
-
The proposed extension in equation (4) is also heuristic (no clear justification for why the gate should be based specifically on high values of or ) and it requires the thresholds to be set. Moreover, RC-A seems to have been proposed based on empirical observations on exactly the same 3,910 benchmark questions which later are used in the main evaluation, RC and RC-A are thus never compared on truly external data.
Questions for authors
Questions/suggestions:
-
Why are and multiplied together? Why not e.g. average them? Is there some kind of probabilistic/mathematical justification for this?
-
"RC-A seems to have been proposed based on empirical observations on exactly the same 3,910 benchmark questions which later are used in the main evaluation, RC and RC-A are thus never compared on truly external data", do you agree with this from above? The evaluation includes two additional LLMs (which demonstrates some form of generalization), but no external datastes/benchmarks?
-
What are the main limitations of your approach? I think this should be at least briefly discussed/mentioned somewhere in the main paper.
-
Based on Appendix C.4 Inference Budget: I think this should be more explicitly evaluated, using fixed-budget baselines. For example, extend Table 2 with versions of "Direct Mean" and "Rubric Mean" that uses V = 8 calls (instead of V = 4), to match the total call budget of RC and RC-A?
-
It's not quite clear to me how the "Paraphrase Mean" baseline method in Table 1 differs from "Direct Mean" in Table 2 (the results are similar but not identical)?
-
Why specifically K=5 verification levels? Based on Figure 11 in the appendix, it seems likely that increasing this beyond K=5 would further improve the overall results?
-
How did you come up with the "Rubric scoring (origin)" prompt in the appendix? Is this based on some previous paper? Why specifically these "Surface validity", "Local correctness", "Logical sufficiency", "Adversarial robustness", "Full verification" as the five levels? And why e.g. "The stated reasoning is sufficient to support the conclusion, without decisive gaps or unjustified leaps" as the prompt for "Logical sufficiency"? At least to me, this all seems a bit arbitrary?
-
Appendix A.5, "For each elicitation type, three additional English prompt paraphrases are generated by an LLM": Could you perhaps give these additional prompts as well? And, which LLM is used to generate these?
Minor things:
- Figure 3: What's to here? Why the same "v" letter for both direct confidence and rubric profile? And, should it be instead of ?
- In Section 4.1, PR-P and PR-N are not explained/defined at all?
- Section 4.2: I would probably mark/format the " only" and " only" in some way in the text here, to avoid confusion.
Clarification of rating:
I quite like this paper overall, I found it quite interesting. But, I think there are a number of clarifications required before it potentially could be accepted.
10 comments
Thank you for the response.
I have read the other reviews and all author responses.
The other reviews are quite similar to mine, I don't think they give me any clear reason to increase my score.
The authors provided a quite solid and comprehensive rebuttal. I think they responded quite well overall. My concerns have partly been adressed, and I would now rate the paper a bit higher than before.
But, I remain quite borderline on the paper. I still think the proposed method seems rather heuristic, and I'm not sure how much impact this would have in practice.
My rating of the paper is higher now than before the rebuttal, but I'm reluctant to increase my score all the way to "3: Weak acceptance". I'm borderline on the paper, leaning slightly towards accept.
Thank you for reading all the reviews and responses and for carefully reconsidering our paper. We genuinely appreciate your recognition of our rebuttal and your explanation of the concerns that remain.
RC is based on an explicit correction principle, which uses rubric support as a response-specific factor to adjust direct confidence: weaker verification support leads to a larger correction. Although RC-A incorporates heuristic design choices, extensive experiments across eight benchmarks and three model families support its effectiveness under a shared default configuration. Table 6 further shows consistent model-averaged gains across neighboring configurations.
Recent studies have demonstrated the practical value of heuristic confidence-guided strategies. Rule-Based Confidence Reasoning [1] uses predefined confidence thresholds for answer selection, while Kumaran et al. [2] show that threshold-based confidence policies govern abstention decisions, with verbalized confidence independently predicting such behavior. DINCO [3] improves verbalized confidence calibration through simple normalization and consistency-based aggregation. These findings suggest that heuristic confidence-based strategies can offer meaningful benefits when empirically validated. Moreover, improving verbalized confidence calibration can support more reliable uncertainty communication and confidence-guided decisions [2, 3], highlighting its potential relevance to both human–AI interaction and model decision-making.
We would be grateful if you could share which specific aspect remains most concerning. If a focused clarification or additional analysis would help strengthen your confidence in the work, we would be very happy to address it. Thank you again for your time and thoughtful consideration.
References
[1] Huang et al. To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts. ICLR, 2025.
[2] Kumaran et al. Causal evidence that language models use confidence to drive behaviour. Nature Machine Intelligence, 2026.
[3] Wang and Stengel-Eskin. Calibrating Verbalized Confidence with Self-Generated Distractors. ICLR, 2026.
Dear Reviewer hbe6,
We hope this finds you well. As the discussion period draws to a close, we would like to respectfully confirm whether we have sufficiently addressed your concerns.
If any questions remain, we would be happy to provide further clarification. Thank you again for your time and constructive comments.
Best regards, The authors