acceptodds
Withdrawn from ICLR 2027

GLIB-SE: Low-Budget Semantic Uncertainty Quantification via Residual Semantic Mass Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable deployment of large language models (LLMs) requires uncertainty estimates that remain informative under tight inference budgets. Existing semantic uncertainty estimators often rely on a limited set of sampled responses to characterize the semantic distribution, which can understate uncertainty when semantically plausible but unobserved alternatives still carry non-negligible probability mass. We propose GLIB-SE, which augments observed semantic classes with a Ghost Cluster to represent residual semantic mass. Semantic counts and energy derived from generated-token scores parameterize an augmented Dirichlet proposal, while response likelihoods constrain Monte Carlo entropy scoring. By incorporating model-internal information into the residual prior, GLIB-SE can distinguish response sets with identical semantic count distributions. We evaluate GLIB-SE using ten responses per input generated by nine language models across six datasets. GLIB-SE achieves higher mean AUROC than likelihood-weighted and count-based semantic entropy in all 18 dataset–model combinations and the highest mean among seven methods in 16 combinations. Support-stratified analysis shows that most of the gain over count-based entropy arises when all sampled answers fall into one observed semantic class, where counts alone provide no ranking information. These findings support combining residual semantic support with model-internal signals to improve error detection under limited sampling.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
uncertainty quantification, large language models, semantic entropy, hallucination detection, residual semantic mass, energy-conditioned priors, Dirichlet modeling

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_DcbsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors build on prior work that attempts to account for missing mass when calculating semantic entropy. At a high level, the method has three main aspects:

  1. A Dirichlet proposal distribution over K+1K+1 classes (the (K+1)th(K+1)^{th} one is the “ghost”). The concentration parameter for the ghost class is based on token scores (Eq. 4-5).
  2. Reject the part of #1 violating the “likelihood-constrained scoring” (Appendix A.1).
  3. Estimate the expectation of the push-distribution induced by the entropy functional with respect to #2 (Eq. 6).

The authors evaluate the method for LLM hallucination detection.

Critical strengths

S1: I think the main outline of the method (at the level I described in the summary) is interesting (though I must caveat with W1 and W3 below). In particular, I think Step 1 - the use of a Dirichlet proposal distribution with token score-based concentration - is a nice idea.

S2: The headline results (Table 1) are promising and appear nicely consistent across scenarios (though I must caveat with W2 below).

Critical weaknesses

W1: At this stage, the writing lacks the clarity and detail necessary for complete evaluation and reproducibility. I highlight:

  1. undefined or underspecified variables (e.g., σ\sigma, SmaxS_{max}),
  2. unexplained modeling decisions (e.g., derivation of Eq. 5, origin of the max(5,⌊M20⌋)max(5, \lfloor \frac{M}{20} \rfloor) threshold, and the fact that the "main configuration" uses γ=10\gamma=10 but γ=1\gamma=1 elsewhere),
  3. unclear descriptions (e.g., Shuffled Energy and the "common rule" for calibrating τ\tau), and
  4. missing experimental details (e.g., prompts for text generation, judging correctness, semantic clustering, etc.).

Please let me know if I have missed any of these.

W2: I have three substantive concerns regarding the evaluation, which limit the ability to contextualize the findings.

  1. The selection of dataset-model combinations is quite confusing. Appendix A.2 indicates 18 dataset-model combinations, as we see in Table 1, but I can't understand how this was decided. In sum, the authors use 99 models and 66 datasets, so I would typically expect 9×6=549 \times 6 = 54 combinations. Can the authors explain where the other 3636 combinations went? Also, I am surprised that a coding model (Seed Coder) is being evaluated on a biomedical question-answering dataset (BioASQ) - and this appears to be the only dataset on which Seed Coder was used.

  2. Given the nature of the method, I find the baselines a bit lacking. I recommend two additions, which I think would be both appropriate and straightforward to include: (1) A strong white-box baseline, such as G-NLL (already mentioned on page 2), and (2) what I’ll call a "basic" Bayesian entropy estimator, which is similar to GLIB-SE with αg=0\alpha_g = 0.

  3. Several statements in the paper suggest that the evaluation may not be consistent or reproducible (perhaps the authors can clarify):

  • "Generation and semantic-grouping backends vary across runs" (Limitations, p. 8)
  • “historical judging prompts are not fully recorded” (Appendix A.2)
  • “The diagnostic’s recomputation path remains to be reconciled with the current experimental implementation” (Appendix A.7)

W3: Analysis of the method (e.g., ablations) is very limited, so it is unclear how the reported improvement arises.

  1. While sensible to include, I wonder if Step 2 in the outline I wrote in the Summary does not do very much (perhaps I am wrong and the authors can illustrate how much is really rejected there), leading me to suspect that Step 1 is the most influential. Since Step 1 is also heavily parameter-dependent, it is crucial for the authors to address the motivation/origin of the parameter choices how sensitive the performance is to them.
  2. I am concerned that the assignment of the entire missing mass to a single class can be problematic when the support size is large relative to the sample size. More broadly, I would be interested in an investigation of some kind of the source of the improvement, especially in comparison to other entropy estimators that account for missing mass. For instance, does Eq. 3 better characterize the missing mass than other missing mass estimators?
Questions for authors

Q1: In Table 2, what do the values for DSE, GLIB-SE, and LNPE represent?


My ratings indicate my perception of the current state of the work, but this is not necessarily reflective of the utility of the method the authors propose, which is difficult to discern in this state; hence, I have tried to be quite specific in my feedback. I look forward to the authors' response, after which I will reassess as appropriate.

Reviewer_yGeHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors present GLIB-SE, an approach to estimating semantic uncertainty from a limited number of sampled responses by introducing an energy-dependent Ghost component for unobserved semantic mass. While the problem is relevant, I have major concerns regarding the methodological contribution, the evaluation, and the reproducibility of the results.

Critical strengths

see above

Critical weaknesses

My main concern is that the authors do not convincingly demonstrate that the proposed residual mass construction provides an advantage over a straightforward combination of semantic counts and token confidence. Their own analysis shows that 77–104% of the improvement over DSE comes from pairs where both examples have only one observed semantic class. In this setting, DSE assigns identical scores, and token-based confidence can resolve these ties. I would therefore have liked to see a simple baseline that uses energy or likelihood to break DSE ties. In addition, the energy-plus-partition baseline basically matches GLIB-SE on GSM8K and TriviaQA and is close on BioASQ and NQ-Open, raising questions about the practical value of the proposed method.

The component experiments are also not convincing. Energy Ghost performs worse than No Ghost on BioASQ, NQ-Open, and TriviaQA, with confidence intervals below zero. Unfortunately, these experiments use different hyperparameters from the main evaluation, so the contribution of the Ghost component under the actual main configuration is unclear. Likewise, the likelihood filter shows no consistent improvement across the six datasets. Taken together, these results do not establish that the central components are necessary.

More conceptually, the authors do not demonstrate that the Ghost component actually estimates unobserved semantic mass. A direct evaluation against missing mass estimated from substantially larger response sets would have been valuable.

I also miss comparisons to the Bayesian semantic entropy estimator of Ciosek et al. [1] and the published Semantic Energy [2]. These are particularly relevant since Bayesian estimation with likelihood constraints and the use of energy to distinguish semantically agreeing responses already exist. Without these comparisons, the incremental contribution remains difficult to assess.

Finally, I find the unresolved scoring discrepancy in Appendix A.7 very concerning. The authors explicitly state that recomputing the full method at the main configuration yields AUROCs 0.0072–0.0200 below the reported results. These differences are comparable to, or larger than, several claimed improvements. This needs to be resolved before the empirical conclusions can be assessed reliably.

[1] Ciosek, K., Felicioni, N., & Ghiassian, S. (2025). Hallucination detection on a budget: Efficient Bayesian estimation of semantic entropy. arXiv preprint arXiv:2504.03579.

[2] Ma, H., Pan, J., Liu, J., Chen, Y., Zhou, J. T., Wang, G., ... & Wang, H. (2025). Semantic energy: Detecting llm hallucination beyond entropy. arXiv preprint arXiv:2508.14496.

Questions for authors

see above

Reviewer_DgfNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed GLIB-SE to solve the gap of semantic entropy underestimating uncertainty with few samples, because unseen meanings get zero mass. The contribution is that it adds a ghost Dirichlet dimension weighted by energy, plus a likelihood filter, and splits questions by K. It is compared with 7 baselines on 6 datasets and 18 dataset-model pairs. Its own appendix says a replay does not match the manuscript.

Critical strengths
  1. The paper reports negative results and uses paired intervals.
  2. The supplement ships the saved scores and code, so the tables can be checked from saved inputs.
Critical weaknesses
  1. There is reproducible issue. The authors’ own replay is 0.0072–0.0200 AUROC below the manuscript. From their CSVs, GLIB-SE is below DSE in 13 of 18 pairs, and K=1 AUROC is about 0.5 on five of six datasets.
  2. The gains are small and the support is limited. Intervals are over questions only (models and seeds fixed), GSM8K labels come from a judge that is also an evaluated model.
  3. The novelty is questionable: a) Combining energy is the main novelty, but the diagnostic has it below No Ghost on three datasets and below Shuffled Energy on two, and A.6 says no independent energy contribution is identified in the main setting. b) Filter is another novelty but Table 7 shows no consistent filter gain. c) Prior work is not run as baseline: Ciosek et al. 2025 (arXiv 2504.03579) already has the Dirichlet posterior, the likelihood-truncated posterior and a learned prior over the number of meanings, and Sun et al. 2026 (arXiv 2603.22812) uses the same truncation, a perplexity-set prior over the number of meanings and a K = Kobs + 1 variant; neither is run as a baseline.
Questions for authors
  1. Can you reconcile the replayed scores with Table 1 and Table 2?
  2. Can you run the energy ablations at the main setting, plus a K=1 tie-breaker.?
  3. Run Ciosek and Sun as baselines, and add intervals over models and seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.