- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper decomposes semantic uncertainty quantification into a comparison operator and an aggregator, and argues that the former is the main bottleneck. The authors train a question-conditioned bi-encoder for factual-equivalence comparison and use it as a drop-in replacement for the operators in SE, CAE, KLE, and COS, reporting consistent improvements across LLM and LVLM benchmarks while reducing pairwise comparison cost from to . The learned representation is also used to improve token-level uncertainty estimation
Critical strengths
-
The operator–aggregator decomposition provides a clear and useful perspective on semantic UQ.
-
The proposed operator is simple and computationally efficient.
-
The experimental coverage is broad.
-
The improvement from replacing the operator is remarkably consistent.
-
The paper includes several useful ablations and diagnostic experiments.
-
The token-level analysis is an interesting byproduct of the learned representation. The highest-weighted token falls inside the ground-truth answer span in 72% of the selected TriviaQA examples, compared with 3.9% under random selection. This suggests that the learned representation captures answer-relevant content beyond its use as a pairwise comparison operator.
Critical weaknesses
-
The training objective does not align with the paper's claims. The authors use correctness labels to construct contrastive learning samples, but answer correctness is not equivalent to semantic equivalence between answers. For example, two incorrect answers may be semantically equivalent or non-equivalent.
-
The motivating “Entailment Trap” example is not convincing as presented. I reproduced the paper’s example, “What is the capital of France?”. When the question is prepended to both answers, which is standard practice in Semantic Entropy and is also explicitly described in Appendix A.2 of this paper, the bidirectional entailment probability is very high (around or above 95% in my test), rather than the low similarity suggested by the motivating example. Appendix A.2 states that each generation is concatenated with before NLI comparison. Therefore, this example does not appear suitable for demonstrating an intrinsic failure of question-conditioned NLI.
-
The primary cause of low entailment appears to be that free-form generated answers contain a large amount of additional factual information, which I do not consider an inherent limitation of the NLI operator. For example, two answers may both correctly state that Thriller was released on November 30, 1982, while adding different claims about release dates in other countries. In such cases, low full-response entailment is not necessarily an NLI failure, because the two complete responses are genuinely not factually equivalent. Following the authors’ setup, I tested 1,000 question–answer pairs from NQ-Open. When the model was instructed to generate short or structured answers, the average entailment score for factually equivalent answers increased to above 0.8. This suggests that a substantial portion of the so-called Entailment Trap can be addressed through generation-side control, without the need to retrain the operator.
-
Most citations in the paper should use \citep{} rather than \citet{}.
Questions for authors
-
Could the authors clarify the exact NLI input construction and scoring rule used for the motivating “Entailment Trap” example and for Appendix F.3? Was the question prepended to both answers exactly as described in Appendix A.2 and in the original Semantic Entropy implementation?
-
Does the proposed operator still provide substantial gains when answer length and output format are controlled? Since concise or structured generation substantially improves NLI entailment in my tests, I would like to see a comparison such as:
- original NLI + free-form generation,
- proposed operator + free-form generation,
- original NLI + concise/structured generation,
- proposed operator + concise/structured generation.
-
How do the authors define “factual equivalence” in the presence of contradictory ancillary information? If two responses give the same answer to the question but contain conflicting additional claims, should they still be considered semantically equivalent?
-
Could the authors provide the trained encoder weights and inference code during the review period?
Thank you for your response. I successfully reproduced the experimental results using [huggingface.co/hugginganonym/Encode_ICLR27], and integrating it into my own method also led to an improvement in the average AUROC. I have updated my score to 3.