- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an evidential framework for uncertainty estimation in autoregressive LLM generation. Sampled responses are grouped into semantic clusters, and transformed token logits are aggregated into cluster-level evidence. The uncertainty score combines vacuity, representing insufficient evidence, with dissonance, representing conflict among clusters. A hyper-opinion extension incorporates semantic overlap through NLI scores and spectral estimates of effective cluster count.
Relative to semantic entropy, the main additions are raw-logit evidence, the vacuity/dissonance decomposition, and overlap-aware scoring. The submission evaluates these components on four QA datasets and three models, with additional translation and sensitivity experiments.
Critical strengths
-
Promising results against a substantial baseline set. Table 1 compares against semantic entropy, graph-based scores, KLE, CoCoA, Semantic Density, EVSE, and internal-confidence methods. For example, Hyper-Sum achieves 77.68 AUROC on Gemma/TriviaQA, compared with 69.44 for SE and 72.57 for the strongest reported baseline. The breadth of comparisons is a strength, although gains are smaller in some settings.
-
Useful investigation of consistently incorrect generations. Appendix D.14 examines single-answer errors across sample counts and compares evidence distributions within the (K=1) subset. Together with ablations on sampling, NLI models, correctness evaluation, and evidence-only scores, this provides empirical support for supplementing semantic dispersion with another signal. Here you could cite [1] to make the claim stronger.
[1] Padhi, Trilok, Ramneet Kaur, Adam D. Cobb, Manoj Acharya, Anirban Roy, Colin Samplawski, Brian Matejek, Alexander M. Berenbeim, Nathaniel D. Bastian, and Susmit Jha. "Calibrating uncertainty quantification of multi-modal llms using grounding." arXiv preprint arXiv:2505.03788 (2025).
Critical weaknesses
- The participation-ratio formula does not satisfy its stated limiting behavior. Equation 7 applies participation ratio to eigenvalues of the normalized graph Laplacian. For complete overlap, taking gives
with eigenvalues . Consequently, , rather than the claimed value of 1. For disjoint self-connected clusters, gives , making the ratio undefined. The singleton case is also undefined. These cases need explicit treatment because they are central to the effective-cluster interpretation.
- The overlap factor cancels in the two-cluster hyper-dissonance formula. For and positive inter-group weights, Eq. 8 simplifies to
At fixed beliefs, the overlap factor therefore has no effect. This needs reconciliation with Figure 12, which attributes a reduction from 0.08 to 0.000158 in a two-cluster example to the inter-group base rate. Effects through the separately changed prior weight should be distinguished from effects of overlap weighting.
-
Raw-logit evidence needs stronger justification. Applying ReLU to raw logits is sensitive to a common additive shift of all vocabulary logits, although that shift leaves token probabilities unchanged. The resulting evidence is therefore not determined solely by the predictive distribution. Its empirical usefulness is interesting, but interpreting it as knowledge or evidential support requires further validation. Proposition 1 bounds assigned belief mass; it does not establish a relationship between that mass and actual correctness.
-
The empirical superiority claims need uncertainty estimates and clearer evaluation details. Some gains over strong baselines are small: Hyper-PR improves AUROC from 73.58 to 73.80 over KLE on Qwen/NQ. Table 1 provides no confidence intervals or repeated-sampling variability. The manuscript also does not clearly specify evaluated dataset sizes or how the target answer is selected from the sampled responses. These details matter for interpreting the margins and reproducing the evaluation.
In summary, the paper offers a potentially useful combination of internal confidence and semantic structure, supported by broad baseline comparisons. My main reservation concerns the consistency of the mathematical formulation, rather than novelty alone. Clarifying whether the identified issues are manuscript errors or implementation issues, and verifying the resulting performance, would help me refine my assessment.
Questions for authors
-
Does the implementation use Laplacian eigenvalues in Eq. 7? How are complete overlap, disjoint clusters, and (K=1) handled?
-
How does Eq. 8 produce Figure 12’s two-cluster reduction when the inter-group weights cancel?
-
How sensitive are the results to logit centering or other probability-preserving shifts?
-
Could you provide dataset sizes, target-answer selection, and paired confidence intervals for the main comparisons?