- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes two methods to turn 3 samples of (class, confidence) tuples per question into an uncertainty score, and measures AUROC. Class here is derived obtained by feeding raw answers into an LLM judge (or a formal equivalence checker), and confidence is prompted from the answering model.
Critical strengths
It’s nice that there are 4 datasets.
Critical weaknesses
- The proposed methods are not really novel or exciting. Essentially Semantic Entropy applied to a specific way of defining semantic equivalence.
- Looking at Figure 6, it looks like the simple “Confidence” baseline (average the confidence statements the model is prompted for) is on par with or outperforms the proposed methods. This is in line with previous works that propose new scores for UQ, where often it does not really beat the baseline.
- Some text is quasi-not understandable and reads AI written, often containing undefined words and being very vague. For example the very first sentence: “A confidence report describes a single response” → what even is a confidence report? This word is used twice in the paper and never defined. Also, this whole paper is not about a single response but question-level uncertainties, aggregating 3 responses. Another example: “Let A be the indices with a parsed answer.”. What indices? i? And what is a parsed answer? Do you mean “We use an LLM judge to map raw answers a_i to predefined classes, and A contains all i where this was successful”? No need to respond to these examples in general, but please rewrite the paper so that it is mathematically and logically precise.
- The eval does not seem to be fair. “LLM entropy [a baseline method, which is never defined neither in main paper nor appendix] groups responses using pairwise equivalence judgments from Claude Sonnet 4.5, the auxiliary model that also supplies the formalizations. This differs from the evaluated Sonnet 4.6 model.”
- The eval uses 100 questions per dataset to generate ROC curves and AUROC, which is way below what is necessary for stable results. Check for example Figure 6 in the appendix, there is no clear trend.
- The paper claims “Once answers and confidences are available, it requires no additional model calls.“ but also implicitly assumes there is some LLM judge mechanism that maps answers to categories, and also that there are meaningful categories to be mapped to in the first place. This mechanism is not described, or even pointed out that it exists.
Questions for authors
None.