- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces a SURF framework for uncertainty estimation in LLM.
Critical strengths
- Some ideas like accounting for the unseen probability mass are interesting and require further investigation.
Critical weaknesses
- The benchmark methods are outdated - the method is compared to SE variants [4,1].
- The evaluation, while utilizing the areas under several different curves is poor, the correctness labels compared against are never specified [2].
- The novelty is non existent - the confidence based approaches exist for a while now and are well described in terms of risk estimation as well as the probability mass arguments [3].
- The clarity is extremely poor. Some sort of optimization is happening in the paper, but the clarity is lacking and I am not sure that it is described in the main paper.
- If it is not evident by this point, the paper fails to address whole lines of work in the field.
References:
- Vashurin, R., Goloburda, M., Nakov, P., Shelmanov, A. & Panov, M. CoCoA: A Generalized Approach to Uncertainty Quantification by Integrating Confidence and Consistency of LLM Outputs. Preprint at https://doi.org/10.48550/arXiv.2502.04964 (2025).
- Ielanskyi, M., Schweighofer, K., Aichberger, L. & Hochreiter, S. Addressing pitfalls in the evaluation of uncertainty estimation methods for natural language generation. in The fourteenth international conference on learning representations (2026).
- Aichberger, L., Schweighofer, K. & Hochreiter, S. Rethinking uncertainty estimation in LLMs: a principled single-sequence measure. in The fourteenth international conference on learning representations (2026).
- Fadeeva, E. et al. LM-Polygraph: Uncertainty Estimation for Language Models. Preprint at https://doi.org/10.48550/arXiv.2311.07383 (2023).
Questions for authors
see weaknesses
2 comments
Clarifying SURF’s Novelty, Baseline Relevance, and Evaluation Methodology (part1)
W3. Novelty of Fragility-Aware Semantic Uncertainty
We respectfully disagree that SURF merely reintroduces existing confidence-based or probability-mass approaches.
The central contribution is not the introduction of confidence scores, semantic consistency, or the classical concept of missing probability mass. Instead, it is the incorporation of finite-sample semantic support fragility into sequence-probability recalibration.
Specifically, SURF characterizes fragility through the proportion of observed semantic probability mass concentrated in singleton clusters, . This quantity acts as an empirical indicator of incomplete semantic coverage and modulates an entropy-regularized optimization that also incorporates local sequence-probability sensitivity.
The cited works address distinct questions:
- CoCoA [1] integrates model confidence and semantic consistency within a minimum Bayes risk framework, but does not introduce singleton-occupancy-based fragility recalibration.
- G-NLL [3] develops a principled single-sequence uncertainty measure based on greedy-decoding likelihood, without explicitly modeling incomplete semantic support under repeated sampling.
- LM-Polygraph [4] provides a comprehensive benchmarking framework rather than the specific finite-coverage correction proposed here.
We do not claim to recover the exact probability mass or identities of unseen semantic clusters. Rather, SURF uses observed singleton mass as a proxy for the instability associated with incomplete sampling.
Our component ablation further supports this distinction: mean AUROC increases from 0.7695 for Rényi semantic entropy to 0.7775 when fragility is incorporated alone, and to 0.7784 when fragility and sequence-probability sensitivity are combined. These results indicate that the fragility component contributes beyond the entropy-only formulation, although its incremental benefit is modest.
W4. Clarity of the Optimization
Contrary to the reviewer's concern that the optimization may not be described in the main paper, the formulation is explicitly presented in the subsection "Objective Function" within Section 3.
The formulation recalibrates each observed token-sequence probability by optimizing three components: (i) a quadratic Rényi entropy term, (ii) a KL-divergence fidelity penalty weighted by sequence-probability sensitivity, and (iii) a fragility-dependent penalty determined by singleton semantic mass.
The parameter controls fidelity to the original sequence probability, while controls the strength of the fragility correction. The recalibrated probabilities are then used to compute semantic-level uncertainty.
Closing Remarks
We appreciate the reviewer's emphasis on rigorous evaluation and engagement with recent literature. Nevertheless, we believe it is important to distinguish the need for broader baseline comparisons and clearer reporting from a demonstration that the proposed method lacks novelty or is fundamentally unsound.
SURF addresses a specific limitation of finite-sample semantic uncertainty estimation: the potential instability of conclusions drawn from incompletely observed semantic response distributions. Our formulation and component ablations provide a concrete contribution toward addressing this limitation.
We hope these clarifications help the reviewer reconsider the methodological contribution of SURF.
Clarifying SURF’s Novelty, Baseline Relevance, and Evaluation Methodology (part1)
We thank the reviewer for their assessment and for highlighting recent developments in uncertainty estimation. We respectfully disagree with the characterization that SURF lacks novelty or that its methodological contribution is unsound. We address the concerns individually below.
W1. Outdated Baselines and Comparison with Recent Methods
We respectfully disagree with the characterization that our benchmarks are outdated. Our evaluation includes Shannon semantic entropy (Farquhar et al., Nature 2024), Rényi semantic entropy and sensitivity-aware recalibration (Vipulanandan et al., ICLR 2026), alongside naive entropy, discrete semantic entropy, embedding regression, and p(True). These are established and recent approaches directly relevant to the problem investigated in this paper.
Furthermore, the reviewer cites LM-Polygraph [4], originally published at EMNLP 2023, with a subsequent benchmarking study published in TACL 2025. While valuable, LM-Polygraph is primarily a framework for implementing and evaluating uncertainty estimation methods, rather than a competing formulation of finite-sample semantic fragility.
We acknowledge that CoCoA (NeurIPS 2025) and G-NLL (ICLR 2026) are relevant recent developments. However, neither addresses the specific methodological problem considered in SURF.
CoCoA (NeurIPS 2025) combines model confidence with semantic consistency through a minimum Bayes risk formulation. Importantly, its authors explicitly acknowledge that consistency estimates obtained from limited samples may fail to capture the full diversity of model outputs. They further identify task-dependent semantic similarity as a limitation, particularly when subtle reasoning errors are not captured by generic similarity metrics. SURF directly investigates a related but distinct issue: the fragility of the observed semantic distribution under incomplete sampling. Rather than relying solely on observed consistency, SURF introduces singleton semantic mass as an explicit signal of finite-sample support instability and incorporates it into probability recalibration.
G-NLL (ICLR 2026) proposes a computationally efficient single-sequence uncertainty measure based on greedy-decoding likelihood. Although theoretically motivated, G-NLL operates at the sequence-probability level and does not explicitly incorporate semantic equivalence or unobserved semantic clusters. Indeed, its authors identify the incorporation of semantic information as an additional direction requiring investigation. Moreover, greedy decoding only approximates the globally most probable sequence, and the method's uncertainty estimates depend on this approximation. Therefore, G-NLL addresses a fundamentally different objective from SURF's finite-sample semantic uncertainty recalibration.
These distinctions are important: combining confidence and consistency, estimating uncertainty from a single sequence, and accounting for incomplete semantic coverage are not equivalent methodological contributions.
Our baseline selection was designed to isolate the incremental effect of fragility beyond established semantic uncertainty formulations. This is supported by our component ablation across 112 paired evaluation settings, where incorporating fragility improves mean AUROC and combining fragility with sequence-probability sensitivity further improves it.
We agree that including CoCoA and G-NLL would broaden the empirical comparison. However, we respectfully maintain that the absence of these additional comparisons does not justify dismissing the existing benchmarks as outdated or concluding that SURF lacks a distinct methodological contribution.
W2. Correctness Labels and Evaluation Methodology
We appreciate the reference to Ielanskyi et al. (ICLR 2026), which identifies important limitations of approximate correctness functions in uncertainty estimation.
Our evaluation distinguishes between the answer being assessed and the generations used to estimate uncertainty: for each question, one response is sampled at temperature 0.1 for correctness evaluation, while ten additional responses are sampled at temperature 1.0 to characterize semantic variability. The same evaluation setting is used for paired comparisons across methods.
We recognize that the manuscript should explicitly state the correctness-labeling function and associated decision criteria, rather than describe only the evaluation metrics. We will clarify this aspect of the experimental protocol and acknowledge the sensitivity to alternative correctness functions emphasized by the cited work.
The results are evaluated across four datasets, seven model variants, and multiple quantization and prompting configurations, producing 112 paired evaluation settings. These comparisons provide evidence of relative performance under the employed evaluation protocol, although they do not eliminate the need for robustness checks against alternative correctness criteria.