acceptodds
Under review as a conference paper at ICLR 2027

When Measures Become Targets: Construct Validity Failures in Uncertainty-Based Hallucination Detection

Abstract

Uncertainty-based hallucination metrics use token confidence, sampled semantic agreement, and representation geometry as proxies for factual reliability. We investigate when these signals diverge from factuality or semantic consistency under controlled perturbations. We develop metric-specific diagnostic tests for First Output Token Entropy, Semantic Entropy, and EigenScore, and evaluate them across four question-answering datasets and four model configurations from three model families. Targeted prompt edits lower first-token entropy while incorrectness is preserved; factual corruption remains below the NLI-based Semantic Entropy threshold; and meaning-preserving paraphrases can increase EigenScore enough to trigger hallucination flags. A human audit supports these discrepancies in the assessed examples and shows that low NLI-estimated entropy can coexist with reduced perceived semantic consistency. We also find related discrepancies under fixed generation, prompting, and rewriting conditions without metric-directed optimization. We therefore identify limitations of the evaluated operationalizations and motivate complementing predictive benchmarks with controlled tests of factuality and semantic preservation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
hallucination detection, adversarial robustness, construct validity, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NPXFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Framed by Goodhart's Law, this paper treats three uncertainty-based hallucination detection metrics (First Output Token Entropy, Semantic Entropy, and EigenScore) as operational proxies for the underlying construct. Through three targeted perturbation modes and a set of ordinary-variation studies, conducted across four model configurations and four QA datasets, the paper shows that metric decisions can systematically diverge from factuality and semantic consistency, demonstrating the consistency of this phenomenon.

Critical strengths
  1. Well-posed research questions with an attack design that accurately reflects them. RQ1 (whether targeted perturbations expose gaps between metrics and human judgements) and RQ2 (whether such gaps also arise under ordinary variation without optimization intent) target the most fragile point in hallucination detection's path from benchmark correlation to deployment reliability, and both problems genuinely exist. Modes 1/2/3 respectively target the structural assumptions of each metric: first-token locality, NLI-based clustering, and representational dispersion, covering both false-negative and false-positive directions; the diagnostic framework and the problem formulation are closely connected.
  2. Restrained claims and clear boundaries. The authors clearly distinguish confabulations from stable errors, score shifts from threshold crossings, and construct-level from implementation-level failures, and repeatedly emphasize that ASR measures proxy–construct divergence under a diagnostic protocol rather than deployment risk, giving the paper a well-delimited line of argument.
  3. Experimental coverage and documentation completeness. The combination of three metrics, four models, and four datasets, the per-mode success criteria, and the human audit with reported IAA statistics give the paper good reproducibility and value.
Critical weaknesses
  1. The models, datasets, and baseline metrics are all dated. The model lineup remains at LLaMA-2-Chat, Mistral-v0.2, and Qwen2-72B; the three attacked metrics, FTE, SE, and EigenScore, are likewise works from several years ago; and the datasets (SQuAD, TriviaQA, NQ, PopQA) are similarly old. I recommend that the authors extend their evaluation to more recent models, methods, and datasets.
  2. The failure analysis proceeds metric by metric, lacking evidence of generalization to other hallucination detection methods. Each of the three modes exploits the specific assumptions of a particular implementation, e.g., FTE's first-token locality, SE's DeBERTa clustering, and EigenScore's middle-layer embedding dispersion, making them essentially three independent attacks. The Goodhart framing implies a general pathology of proxy metrics, yet the evidence covers only three points within the uncertainty-based family. The paper offers no experiments or arguments on whether retrieval-based, model-based, or supervised detection methods are equally fragile.
  3. A circular NLI dependency runs through the entire evidence chain: the construct's ground truth and the audited metric share the same failure modes. DeBERTa-large-MNLI plays four roles in the paper: it is the clustering component inside SE (the target of Mode 2), the Stage-2 factual-correctness oracle for all three modes, the arbiter of the semantic-equivalence/single-cluster constraint in Mode 3's success criterion, and the candidate filter for the Mode 2 attack. This means that in Mode 2, the determination that "the facts have been corrupted" and the observation that "SE failed to detect the corruption" come from the same model. Attack success may merely reflect correlated NLI errors across the two stages rather than a genuine divergence between the metric and the construct. The human evaluation was the right idea as an independent check, but inter-annotator agreement on ES consistency is near zero — the point where independent evidence is most needed is precisely where it is least trustworthy.
Questions for authors

See Above.

Reviewer_iKeeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether uncertainty-based hallucination detectors measure factuality, or only proxies that can diverge from it. The authors study three detectors for short-form question answering, First Output Token Entropy, Semantic Entropy, and EigenScore, and design one perturbation for each that changes the detector's decision without the corresponding change in correctness or meaning. They also measure how often similar mismatches occur under ordinary generation and routine prompt or wording changes. The paper does not propose a new method or a theoretical result. Across four models and four QA datasets, the targeted perturbations succeed in every model and dataset combination, and a human audit supports the mismatches on a sample of examples.

Critical strengths

The experiments support the claims. For each of the three detectors, the targeted perturbations change the detector's decision in every model and dataset combination, and the paper carefully separates score shifts from threshold crossings when reporting these results. The evaluation also covers four models from three families and four QA datasets, with a human audit of 150 examples per detector.

Critical weaknesses

While the experiments are carried out carefully, I have three concerns about what the paper contributes beyond existing work and about whether its tests examine the detectors under the conditions they are designed for.

  1. The main finding is largely known from prior work.

That uncertainty-based detectors are imperfect proxies for factuality is already established. The paper itself cites Janiak et al. (2025) [1], who show that simple length heuristics can rival these detectors and that their reported performance depends heavily on the evaluation metric, and Kulkarni et al. (2025) [2], who find that hallucination detection metrics often fail to align with human judgments and to generalize across datasets. Zeng et al. (2024) [3] further show that the uncertainty of an LLM can be manipulated while its output is kept fixed, which is closely related to the targeted perturbations here but is not discussed. The ordinary variation study also adds little beyond standard evaluation. How often a detector fails to separate correct from incorrect answers under ordinary generation is what standard uncertainty metrics such as AUROC and the prediction rejection ratio already measure [4, 5], whereas the reported transition rates, for example the 28.2% of incorrect answers missed by Semantic Entropy, depend on a threshold chosen on the same evaluation data. It is therefore unclear what the paper shows beyond the known fact that these proxies are imperfect.

  1. The targeted tests for Semantic Entropy and EigenScore operate outside the conditions these detectors are defined for.

Both detectors are defined over a set of responses sampled from the model for the same question, and their signal comes from how much these samples agree. Modes 2 and 3 replace this set with responses that the model did not generate. Mode 2 substitutes key answer tokens with embedding neighbours, producing strings such as "President Enable" and "President Everyone" (Figure 1), and Mode 3 searches over rewrites for the set with the highest dispersion. A detector built on the model's own sampling distribution is not expected to stay valid on such inputs, so these results show what happens when the sampling assumption is broken, rather than a failure of the detector under the conditions it is meant for. The evidence that the rewrites preserve meaning is also limited, since the main text notes low inter-annotator agreement on EigenScore consistency, and the automated checks share an NLI component with the correctness check.

  1. The scope is too narrow to support conclusions about uncertainty-based detection in general.

Uncertainty-based hallucination detection includes many methods that differ in whether they need sampling, access to logits or internal states, and training, such as sequence likelihood measures, graph-based consistency measures, verbalized confidence, and supervised probes [4]. The paper examines three of them, all on short-form QA, and on models that are now dated (LLaMA-2-Chat 7B and 13B, Mistral-Instruct-v0.2 7B, and Qwen2-Instruct 72B). Evaluating more models of a similar kind does not address this, because the findings follow from the design of each detector rather than from the model. It remains unclear whether the conclusions apply to the broader family of methods named in the title.

[1] Janiak et al., "The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs"

[2] Kulkarni et al., "Evaluating Evaluation Metrics: The Mirage of Hallucination Detection"

[3] Zeng et al., "Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models"

[4] Kang et al., "Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directions"

[5] Vashurin et al., "Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph"

Questions for authors
  1. (Standard evaluation) Could you report AUROC and the prediction rejection ratio of the three detectors on the ordinary generations, so that the ordinary variation results can be compared with the standard evaluation of these methods?
  2. (Sampling assumption) For Semantic Entropy and EigenScore, how do the detectors behave when only the prompt is perturbed and the response set is still sampled from the model, as in Mode 1?
Reviewer_dGcfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper shows that various hallucination detection techniques can fail in unique ways, and shows how to attack them to evade them. It studies first output token entropy, semantic entropy, and eigenscore, for hallucination detection, and shows how to construct cases where these techniques fail to detect factuality.

Critical strengths
  1. The technical work in the paper is good, and I believe it can be valuable, although it should be noted that similar work has been done before in the literature. Yet, it does provide new insights.
Critical weaknesses
  1. I don't believe the central pillar of the paper is correct: that any of the methods studied in the paper are hallucination metrics. They are hallucination detection methods, and they are tested against facts in benchmarks (either in direct QA settings or with an LLM as a judge for generated outputs). You cannot just call a detection technique a 'metric' and then argue why the metric doesn't capture the true construct. Of course it doesn't, it's a detection technique, not a metric! In all the papers I've read about hallucinations in LLMs, and even the references provided in this paper itself, no other work calls it a metric or tries to optimize for it, so it's unclear to me where this narrative is coming from.
  2. The technical contributions, while good, should be well placed in the existing literature, and I don't believe they are. For instance, first output token entropy is not a good measure of the eventual model's output and is easily influenced by prompt perturbations has been shown before (Wang et al., 2024; Shao et al., 2024), similarly the argument that uncertainty-based hallucination detection techniques don't really measure correctness has also been present in the literature (Ganesh et al., 2026). It is important to clarify what novelty this paper brings to the table, and how it is different from existing literature.
  3. The mathematical notation needs some work. For instance, y(x) is never defined, which I assume from context is probably the LLM generation? Similarly, the target construct 'C' is used freely as a variable, as a function, as a function with a subscript, etc. It is important to define what all these uses mean.

References ->

  1. Wang, Xinpeng, et al. "“My answer is C”: First-token probabilities do not match text answers in instruction-tuned language models." Findings of the Association for Computational Linguistics: ACL 2024. 2024.
  2. Shao, Justin. "First Token Probabilities are Unreliable Indicators for LLM Knowledge." (2024).
  3. Ganesh, Prakhar, Reza Shokri, and Golnoosh Farnadi. "Rethinking hallucinations: Correctness, consistency, and prompt multiplicity." Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
Questions for authors
  1. What papers have used hallucination detection methods as a metric and have tried to optimize for it? Can you establish a trend of this happening in the literature?
  2. What are your technical contributions in the context of the existing literature, given the new references I have shared above? Moreover, I would recommend a more thorough inspection of related works, to see whether other techniques have also been shown to fail in standard ways before.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.