acceptodds
Under review as a conference paper at ICLR 2027

UAHV: Uncertainty-Aware Hierarchical Detection for Hallucination Detection in Large Language Models

Abstract

To mitigate hallucinations in large language models (LLMs), recent methods have increasingly adopted fine-grained decomposition or repeated reasoning to improve factual verification. However, they often apply predefined verification procedures across samples, overlooking the fact that factual conflicts can differ in scope, evidence requirements, and decision uncertainty. This mismatch may lead to either missed conflicts or erroneous factual judgments. To address this limitation, we propose Uncertainty-Aware Hierarchical Verification (UAHV), an adaptive framework that adjusts verification according to sample-specific factual states. UAHV introduces two complementary mechanisms: Hierarchical Factual Verification (HFV) progressively routes samples from global semantic screening to local verification and selectively invokes arbitration for uncertain cases, while Type-Aware Asymmetric Verification (TAV) applies different evidence-matching strategies to evidence-sensitive and semantically flexible facts. Experiments on three datasets and four LLM backbones show that UAHV achieves the best F1 score in 11 of 12 settings, with an average F1 of 76.12%, outperforming the strongest baseline by 6.27 percentage points.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Hallucination detection, Large Language Models, Hierarchical Verification, Uncertainty-Aware Reasoning, Type-Aware Verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ibGnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hallucination detection and proposes UAHV. It combines global LLM screening with local lexical and semantic verification, as well as additional LLM verification for borderline cases. It aims to improve detection performance while reducing LLM calls.

Critical strengths
  1. UAHV achieves the highest reported macro-F1 among the evaluated methods in 11 of the 12 dataset–model settings.
  2. The ablation results provide support for the method’s main design choices.
Critical weaknesses
  1. The novelty is modest, and the verification rules need further justification. For example, HFV cannot correct false positives after the global screening. Errors missed globally may also be accepted when too few keywords remain unsupported to trigger arbitration, even if those keywords represent a critical fact.
  2. Key implementation details are unclear. The paper does not clearly explain keyword extraction or common-noun identification. Although search ranges and sensitivity results are provided, it remains unclear how the default fuzzy matching threshold of 0.75 and missing-count threshold of 4 were selected.
Questions for authors
  1. Please explain how keywords are extracted and common nouns are identified, and how the default fuzzy matching threshold of 0.75 and missing-count threshold of 4 were selected.
  2. In Table 2, adding rescue increases SUM FPR from 16.60% to 28.00%. As defined in the paper, rescue can only reduce or maintain the missing count, so FPR should not increase in this setting. Please explain this result.
  3. Please report how often the global stage incorrectly rejects supported responses and how much these errors contribute to the final false-positive rate.
  4. How well does the fixed missing-count threshold work across samples with different numbers of keywords? How does UAHV handle a response with one incorrect critical fact when the remaining facts are supported?
2 comments
AuthorsOpenReview ↗

Request for Further Feedback on Our Rebuttal

Dear Reviewer ibGn,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them). Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

AuthorsOpenReview ↗

A Brief Follow-up Before the Discussion Closes

Dear Reviewer ibGn,

We apologize for following up once more, as the discussion period will close in a few hours. We have carefully addressed all of your comments and would greatly appreciate any brief feedback on whether your main concerns have been resolved. If so, we would also be grateful if you could reconsider your evaluation accordingly.

Thank you again for your time and thoughtful review.

Reviewer_AfAfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes UAHV, an adaptive framework for reference-grounded hallucination detection. Hierarchical Factual Verification first applies a response-level global verifier, which can terminate with REFUTES. Passing pairs undergo local verification based on unsupported keywords; common-noun keywords that fail lexical matching may receive semantic rescue through embedding similarities to evidence. Cases near the local decision threshold are escalated to an evidence-constrained LLM arbiter. Across HaluEval-Summary (2,000 instances), QAGS-CNNDM (233), and FEVER (500), using four backbones, UAHV achieves the best reported macro F1 in 11 of 12 settings, averaging 76.12%, or 6.27 percentage points above the strongest baseline on average. A separate efficiency study with Llama-3.1-8B reports 1.03–1.35 generative LLM calls per sample and lower latency than G2LDetect. The appendices provide component ablations, sensitivity analyses, boundary-uncertainty measurements, and case studies. The correspondence between the main-text and appendix ablation configurations needs clarification before drawing an overall component attribution.

Critical strengths
  1. Broad evidence for the routing approach. The reported macro-F1 advantage in 11 of 12 dataset–backbone settings supports the usefulness of combining global screening, local checks, and selective arbitration across multiple models. Boundary analyses also find higher error rates near the threshold, providing empirical motivation for escalation, although the small boundary subsets on QAGS and FEVER warrant caution.
  2. Useful efficiency and component measurements. Appendix A.4 measures efficiency on 100 examples per dataset with Llama-3.1-8B at batch size 1. In this setting, UAHV uses 1.03–1.35 generative calls per sample versus 3.11–6.94 for G2LDetect and is approximately 4.3–10.3 times faster. This is evidence for a favorable trade-off relative to G2LDetect, while SelfCheckGPT has lower absolute latency. The appendix also reports both corrected and harmed predictions from semantic rescue, allowing its limited marginal benefit in that setting to be assessed.
Critical weaknesses
  1. Local verification and component attribution need clarification. Local elements are deduplicated, POS-filtered keywords. Appendix A.4 mentions additional relational, polarity, and binding constraints, but their implementation is insufficiently specified to assess how broader factual inconsistencies are checked. Moreover, the ablation results are not readily reconcilable: on SUM, Table 2 reports macro F1 of 0.6857→0.6994 when adding semantic rescue without arbitration, whereas Appendix Tables 6–7 report 0.6776→0.6785 for the corresponding description. These may use different configurations, but the mapping is unclear. The appendix supports a larger marginal effect from arbitration than rescue in its reported setting; it does not establish that arbitration accounts for most of the full method's advantage over baselines.
  2. Uncertainty, operating-point trade-offs, and model selection are incompletely characterized. Table 1 lacks paired significance tests or confidence intervals, which matters for small gaps on the 233-example QAGS set. SUM false-positive rates range from 14.60% to 34.80% across the four backbones. Some lower-FPR baselines have worse macro F1, so this is a trade-off requiring application-specific interpretation rather than evidence of uniform inferiority. Finally, the split used to select the Table 3 defaults is not clearly specified; this prevents assessment of model-selection independence, without itself establishing test-set tuning.
Questions for authors
  1. On which split were the Table 3 defaults selected, and were the final evaluation examples excluded from all hyperparameter selection?
  2. Please reconcile the SUM and QAGS ablation values in Table 2 and Appendix Tables 6–7, including the backbone, sample set, thresholds, and component settings used in each.
  3. Appendix A.4 includes relational, polarity, and binding checks in the O(ML) complexity term. Where are these checks implemented, and can their incremental contribution be isolated?
  4. Table 1 already includes a Standard response-level baseline. Does it use exactly the same prompt and settings as the global gate G? Please clarify or provide a matched comparison of G alone, G plus the local stage, and the complete system.
  5. Can you report paired uncertainty for the Table 1 improvements and characterize the false-positive/recall trade-off at matched operating points?
Reviewer_APFsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UAHV, an uncertainty-aware hierarchical verification framework for hallucination detection. It combines global semantic screening, local factual verification, and uncertainty-triggered arbitration. It also introduces type-aware asymmetric verification to handle different factual elements. Experiments on three datasets and four LLM backbones show improvements over existing baselines.

Critical strengths
  1. The paper focused on an important problem in LLM hallucination detection.
  2. The hierarchical verification framework is well-motivated and clearly presented.
  3. The experiments over multiple datasets, LLM backbones, and baselines are competitive.
Critical weaknesses
  1. The proposed framework mainly combines global verification, keyword matching, semantic similarity, and LLM-based arbitration. The technical contributions beyond existing verification methods are relatively incremental.
  2. TAV mainly relies on part-of-speech information and lexical matching. It is unclear how the method explicitly handles relational consistency, predicate polarity, and factual bindings, as claimed in the paper.
  3. The uncertainty signal is based on proximity to a predefined missing-count threshold. So the evaluation of uncertainty-aware routing is insufficient.
  4. The appendix shows only marginal improvements from semantic rescue, with no improvement on QAG. This raises questions about the effectiveness of TAV as a key component.
  5. Table 2 and the appendix report different results for seemingly identical experimental settings. For example, the SUM results for Global + Lexical and + Rescue differ from Table 2, 6 and 7. These inconsistencies need to be further clarified.
Questions for authors

See my weakness.

4 comments
AuthorsOpenReview ↗

Request for Further Feedback on Our Rebuttal

Dear Reviewer APFs,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them). Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Thank the authors for the detailed rebuttal and additional experiments. The responses have addressed some of my concerns regarding the inconsistent ablation results and uncertainty-aware routing. However, my concerns about the limited technical novelty remain. In particular, the independent contribution of TAV is still unclear, and the additional ablation results show that some structural checks can lead to lower performance on SUM and QAG. These issues need further investigation. Therefore, I will maintain my original score.

AuthorsOpenReview ↗

Response: Clarification of TAV's Contribution, Structural Verification, and Methodological Novelty

Thank you for your continued feedback. We further clarify TAV's distinct contribution, the interaction between structural verification and boundary arbitration, and UAHV's evidence-state-driven novelty.

(1) TAV's contribution. The Global + Lexical configuration retains type-aware fact identification and structural checks, with Semantic Rescue and Arbitration disabled. Thus, its comparison with +Rescue isolates Semantic Rescue rather than TAV as a whole.

ConfigurationArbiterSUMQAGFEVER
G-onlyOFF0.59840.76160.7240
Global + LexicalOFF0.67760.75820.7573
+RescueOFF0.67850.75820.7633
↳ w/o RelationOFF0.69460.76220.6919
↳ w/o PolarityOFF0.69400.76220.6919
↳ w/o BindingOFF0.67470.76160.7633
+Arbiter (Rescue OFF)ON0.70010.76160.7663
Full UAHVON0.70470.77110.7757

Macro-F1 on LLaMA-3.1-8B-Instruct. Structural ablations are relative to +Rescue, with Arbitration disabled.

Compared with G-only, adding complete type-aware local verification improves Macro-F1 by 8.01 pp on SUM and 3.93 pp on FEVER; on QAG, Full UAHV achieves a 0.95 pp gain over G-only, following a 0.34 pp decrease at the non-arbitrated local stage. Furthermore, on 117 controlled common-noun cases, TAV improves valid acceptance from 88.46% to 92.31% while reducing false acceptance from 41.76% to 28.57%, demonstrating its selective semantic-verification capability.

(2) Structural verification and arbitration. To directly address your concern regarding Relation and Polarity on SUM and QAG, we conducted additional controlled ablations with Arbitration enabled.

Configuration (Arbiter ON)SUMQAGFEVER
Full UAHV w/o Relation0.68650.76630.7045
Full UAHV w/o Polarity0.69190.76630.6907
Full UAHV0.70470.77110.7757

Removing Relation reduces Macro-F1 by 1.82/0.48/7.12 pp, while removing Polarity reduces it by 1.28/0.48/8.50 pp on SUM/QAG/FEVER.

Crucially, the contribution of structural checks reverses when Arbitration is enabled. On SUM, removing Relation changes from a +1.61 pp gain without Arbitration to a −1.82 pp loss with Arbitration; removing Polarity similarly changes from +1.55 to −1.28 pp, with the same reversal on QAG (+0.40 to −0.48 pp for both checks).

Abstractive paraphrasing in SUM/QAG may introduce dependency mismatches during local verification, whereas removing structural constraints may overlook genuine relational or polarity conflicts. These matched ablations demonstrate that structural verification and boundary arbitration interact in determining final performance: TAV preserves fact-sensitive inconsistency signals, while HFV uses the resulting evidence state for local decisions and selective arbitration.

(3) Methodological novelty. UAHV transforms fact-type-dependent evidence judgments into signals governing subsequent verification decisions, rather than applying independent verification operators sequentially.

Semantic Rescue yields Macro-F1 gains of 0.09/0.00/0.60 pp without Arbitration versus 0.46/0.95/0.94 pp with Arbitration, corresponding to positive interaction gains of 0.37/0.95/0.34 pp on SUM/QAG/FEVER. Under equal arbitration budgets, boundary-guided routing further outperforms random routing by 1.63/0.95/1.30 pp, respectively.

Together, these results substantiate UAHV's central methodological contribution: coupling TAV's asymmetric factual evidence verification with HFV's evidence-state-driven adaptive decisions, rather than merely combining existing verification techniques.

AuthorsOpenReview ↗

A Brief Follow-up Before the Discussion Closes

Dear Reviewer APFs,

We apologize for following up once more, as the discussion period will close in a few hours. We have carefully addressed all of your comments and would greatly appreciate any brief feedback on whether your main concerns have been resolved.

Thank you again for your time and thoughtful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.