acceptodds
Under review as a conference paper at ICLR 2027

AWARD: Answer-Level Weighted Aggregation of Retrieval-Free Diagnostics for Hallucination Detection

Abstract

Hallucination in large language model (LLM) outputs is often flagged by inference-time signals that are then used directly as answer-level scores, despite measuring a different quantity. This paper presents AWARD, a retrieval-free hallucination detector that separates a formal sequential source diagnostic from a supervised answer-level score. The core signal is Speculative Cross-Model Divergence (SCMD), the pointwise log ratio between an instruction-tuned model and its own pretrained base. Under the base-sampling null, the SCMD product is an e-process satisfying Ville's anytime type-I bound. SCMD is one of four inference-time primitives, joined by primary-model entropy, Attention-Grounding Entropy (AGE), and Semantic Refutation via NLI (SRN). Across five model families and three benchmarks (15 cells), nested cross-validation yields mean AUROCs of , , and on HaluEvalQA, MatSciQA, and TruthfulQA, respectively. In comparisons of 10-seed paired-bootstrap with the strongest of DoLa, SelfCheckGPT, and Semantic Entropy, AWARD achieves mean , with positive differences in cells; two survive Holm–Bonferroni correction in family . AWARD wins cells with mean against the stronger of two classifier- and label-matched supervised fusions. Split-conformal calibration provides marginal false-positive-rate control under exchangeability. At the lowest non-degenerate tier, , cells meet the seed-averaged target and meet an item-clustered upper-confidence-bound criterion; the tier is degenerate in every cell because the calibration sets are too small. All four primary feature blocks have positive mean contributions in leave-one-block-out ablations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hallucination detection, large language models, retrieval-free detection, uncertainty estimation, conformal calibration, e-processes, supervised score fusion.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bcwwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes AWARD, a retrieval-free hallucination detector combining cross-model divergence, entropy, attention statistics, and NLI-based signals through supervised aggregation.

Critical strengths

The paper provides a relatively rigorous empirical evaluation, including nested cross-validation, matched supervised baselines, statistical significance testing, ablation studies, and cross-model transfer analysis.

Critical weaknesses
  1. Limited Methodological Novelty. AWARD primarily combines existing hallucination detection signals using standard supervised classifiers. The proposed SCMD statistic is based on an existing likelihood ratio, while the remaining components largely adapt established techniques.

  2. Disconnect Between Theory and Practice. The theoretical guarantee applies only under a base-model sampling assumption, whereas the experiments use greedy decoding from instruction-tuned models. Furthermore, the guarantee concerns distributional divergence rather than factual correctness, making its relevance to hallucination detection unclear.

  3. Weak Practical Detection Performance. The method achieves relatively modest AUROC values, and its mean detection power is only 9.9% under a 5% false-positive-rate target. This raises concerns about its practical effectiveness.

  4. Insufficient Generalization Evidence. Experiments primarily involve older, relatively small language models. Cross-family transfer also shows performance degradation, limiting confidence in generalization to modern LLMs.

  5. Potential Evaluation Bias. The NLI model used to construct semantic refutation features also contributes to TruthfulQA correctness labels. Although an independent judge is tested, this shared dependency raises concerns about evaluation validity.

Questions for authors

NA

Reviewer_p7MiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AWARD, a retrieval-free hallucination detection method for LLMs. It combines four signals: Speculative Cross-Model Divergence (SCMD), model entropy, Attention-Grounding Entropy (AGE), and Semantic Refutation via NLI (SRN). These features are aggregated using a supervised classifier to produce answer-level hallucination scores. The paper also provides a theoretical analysis. Experiments across five backbone models and three benchmarks show improvements over several existing baselines.

Critical strengths
  1. The paper targets an important problem in LLM hallucination detection.
  2. The method combines multiple complementary signals without requiring external retrieval.
  3. Experiments across five backbone models and three benchmarks show improvements over several baselines.
Critical weaknesses
  1. SRN relies on fixed refutation templates whose relationship to factual correctness is unclear. TruthfulQA labels are also generated using the same NLI model used by SRN. Although there's a independent-judge experiment, I still have concern with potential label bias. The filtering process that leaves only 114–172 labeled outputs per model also needs clarification.
  2. Only 2 of the 15 comparisons remain significant after Holm–Bonferroni correction. The cross-family transfer evaluation reuses benchmark questions and does not strictly hold out entire model families. In addition, the implementations of the BF and CBF baselines and some model-specific experimental settings are not clearly described. These missing details make it difficult to assess the fairness and reproducibility of the experiments.
  3. At a target FPR of 0.05, the mean detection power is only 0.099. Most incorrect answers remain undetected. Comparisons with baselines under the same FPR constraints would be more informative.
  4. The SCMD e-process guarantee only holds under the base-sampling null. It does not apply to the greedy decoding used in the experiments or directly guarantee factual correctness. Its contribution to the actual hallucination detector is therefore unclear.
Questions for authors

See my weakness.

Reviewer_VcF6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes AWARD, a retrieval-free hallucination detector that combines divergence between an instruction-tuned model and its pretrained base with entropy, attention statistics, and NLI-based features. A supervised classifier produces an answer-level score, and split-conformal calibration sets the detection threshold. The paper also establishes an anytime-valid guarantee for the divergence process under a base-sampling null, while distinguishing this guarantee from factual correctness and the deployed detector.

Critical strengths

Combining complementary model signals is reasonable, and the experiments show that fusion improves detection. The nested validation and label-matched supervised baselines help address whether the gains come simply from supervision. I also appreciate the reporting of uncertainty, multiple-comparison correction, and limitations, including low detection power and the restricted scope of the theoretical guarantee.

Critical weaknesses
  1. The manuscript is hard to follow and read, as many parts of the manuscript seem directly written by AI tools.

  2. The connection between the theoretical result and the implemented detector is weak. For example, the implemented detector is a supervised feature fusion whose correctness-related guarantee comes from a separate, standard calibration procedure. What the theoretical result contributes beyond motivating an empirical feature remains unclear.

  3. The interpretation of SRN as semantic refutation is not well supported, i.e., an incorrect factual statement does not normally entail a statement declaring itself false.

  4. The TruthfulQA evaluation has unresolved dependence between features and labels. The same NLI model contributes detector features and assigns correctness labels, potentially rewarding agreement with the labeling model rather than factual accuracy.

Questions for authors

Please see the weakness part and indicate to what extent using LLM for manuscript writing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.