- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper conducts a large-scale benchmarking study across 19 english-output datasets and 13 LLMs, as well as 12 model translation experiments. Their experiments explore several semantic uncertainty methods from the literature, compared against three other sampling-based baselines that do no use NLI-style scoring. They conduct descriptive analysis, including looking at false positives (instances where UQ method predicts correctness but actual answer is wrong or low-quality) as well as correlations of confidence vs. model size and correctness.
Critical strengths
- The paper is extremely thorough in its model and dataset breadth, covering 19 english-datasets and 13 LLMs.
- Quality of writing is generally very good. It is very readable prose and did not require any extra effort to understand the methods or setup.
- When UQ methods fail is an important research question
Critical weaknesses
1) The benchmarking study is not sufficiently distinguished from prior benchmarking studies (Vashurin et al., TACL 2025, Zhang et al., TMLR 202, Bouchard et al., TMLR 2025, Bakman et al., ACL 2025). In particular, the LM-Polygraph TACL study is of similar breadth, but investigates many more UQ methods. 2) The descriptive analysis is shallow and does not yield any meaningful insights. SURE-OC, the headline methodological contribution (which the authors honestly discuss as a diagnostic), is effectively just a coined term for false positives (instances where a UQ method predicts correctness but the model got it wrong / answered with low quality). Coining "overconfidence" is more tautological than insightful and is not itself a contribution. The correlations show that UQ scores correlate with correctness/quality, but this has shown by most previous studies on the topic. 3) Method correctness and/or clarity concerns:
- LUQ-pair is used on math questions, but the claim/sentence-level decompositions do not make sense in that context
- the authors appear to conflate discrete scoring as discrete in NLI judgements vs. discrete in cluster probabilities (discrete semantic entropy). The writing should be more clear about this 4) Incomplete experiment description in main text: The authors should clearly define which scorers they are using, instead of just citing the reference papers. The reader has to deduce purely from acronyms in the figure. The main text also does not mention which LLMs are used (reader again must deduce from figure). 5) Weak baselines used for long-text setting: UQ for long-text outputs has been well-established, but the authors only include a single method for this setting (LUQ-PAIR). From a benchmarking study, one would expect several more baselines that operate at the claim level, such as LUQ-atomic (Zhang et al., EMNLP 2024; Zhang et al., AACL 2025), Long-form semantic entropy (Farquhar et al., Nature 2024), claim-level graph-based methods (Jiang et al., NeurIPS 2024), and generalizations of these (Bouchard et al., TMLR 2026).
References:
- Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph. Vashurin et al., TACL 2025: Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
- Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers. Bouchard et al., TMLR 2025: https://openreview.net/forum?id=WOFspd4lq5
- Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models. Lin et al., TMLR 2024: https://openreview.net/forum?id=DWkJCSxKU5
- Reconsidering LLM Uncertainty Estimation Methods in the Wild. Bakman et al., ACL 2025: https://aclanthology.org/2025.acl-long.1429/
- LUQ: Long-text Uncertainty Quantification for LLMs. Zhang et al., EMNLP 2024: https://aclanthology.org/2024.emnlp-main.299/
- Atomic Calibration of LLMs in Long-Form Generations, Zhang et al., AACL 2025: https://aclanthology.org/2025.findings-ijcnlp.9/
- Graph-based Uncertainty Metrics for Long-form Language Model Generations. Jiang et al., NeurIPS 2024: https://proceedings.neurips.cc/paper_files/paper/2024/hash/39c5871aa13be86ab978cba7069cbcec-Abstract-Conference.html
- Detecting hallucinations in large language models using semantic entropy. Farquhar et al., Nature 2024: https://www.nature.com/articles/s41586-024-07421-0
- Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study. Bouchard et al., TMLR 2026: https://openreview.net/forum?id=gngp4Zz9Sj
Questions for authors
- Why did you only use LUQ-pair, despite having long-text outputs for some datasets?
- How does LUQ-pair work for math questions?
- Can you add the model names and scorer names to the main text?
- What does your benchmark study contribute beyond existing UQ benchmark studies?