- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes VSI, a training-free method for detecting hallucinations in LVLMs. It uses VSC to measure how coherent the attended visual tokens are, and VSR to measure the relative visual vs. textual contribution during decoding. The two signals are combined to score whether a generated object is visually grounded. The method is evaluated across multiple LVLM architectures and hallucination benchmarks.
Critical strengths
-
The paper provides a simple and interpretable way to separate visual representation quality from visual influence during decoding.
-
The method is training-free and can be applied across different LVLM architectures without additional model training.
-
The experiments cover multiple models and hallucination settings, with ablations showing that VSC and VSR provide complementary signals.
-
The paper is generally easy to follow.
Critical weaknesses
This is a well-made paper. However, I still have strong concerns regarding the novelty, soundness and evaluation quality of this work.
-
The motivation behind both observations is not sufficiently convincing. For VSC, comparing object-related visual tokens with randomly sampled tokens is expected to produce higher similarity, since tokens belonging to the same object already share appearance, texture, and spatial context. This does not demonstrate that hallucinated objects should have less coherent visual representations. For VSR, the POPE intervention mainly introduces an explicit instruction bias by telling the model that an object exists or does not exist. This is different from the language priors usually discussed in hallucination work, such as object co-occurrence or attribute priors, so the experiment does not strongly support the claimed language-prior interpretation.
-
The novelty and positioning against recent work are unclear. VisER[1] already decomposes object hallucination detection into visual evidence and visual reliance, which is conceptually very close to the proposed VSC/VSR formulation. EAZY[2] and DAMRO[3] are exactly two counter examples for the observation of this paper that "the attention of hallucinated objects is more sparse than real object." Especially, EAZY[2] is also a training-free object hallucination detector and shows that hallucinations can be driven by highly attended image tokens, including cases where attention is concentrated on incorrect but visually similar regions. This suggests that strong or concentrated visual signals do not necessarily imply correct grounding. AdaIAT[3] further shows that stronger attention to previously generated text can reduce hallucination, suggesting that textual contribution should not be directly interpreted as harmful language prior. These works should be discussed in much greater depth, and VisER and EAZY should be included as direct comparisons.
-
The mechanistic claims are stronger than what the paper actually establishes. The paper does not statistically demonstrate that hallucinated generations generally rely on more semantically dispersed visual regions, nor does low VSR uniquely indicate that language priors override visual evidence. VSR itself is still derived from the attention sublayer and measures the visual and textual components of the attention-induced residual update, rather than the full residual stream or the final causal contribution to the output. Therefore, the proposed metrics may serve as diagnostic signals, but the current experiments do not establish the claimed failure mechanisms.
-
The distinction from prior attention- and representation-based methods is somewhat overstated. Both VSC and VSR are constructed from attention weights and hidden representations, and VSR can be viewed as a more detailed attention-output contribution measure rather than a fundamentally different source of information. Stronger ablations against simpler attention- and representation-based signals are needed to show what is specifically gained by the proposed formulations.
-
The theoretical analysis is limited. The propositions are mainly algebraic reformulations: VSC rewrites a weighted pairwise similarity as a squared norm, while VSR decomposes the attention output according to source tokens. These identities may be useful computationally, but they do not constitute substantial theoretical contributions by themselves.
-
The evaluation has important limitations. Both MSCOCO and Objects365 rely on fixed object taxonomies and CHAIR-style matching, which can introduce noisy labels for open-ended generations due to missing categories, coarse ontology mappings, and incomplete semantic coverage. Objects365 expands the vocabulary but does not remove the underlying issue. Given that some of the improvements over baselines are relatively small, it is difficult to determine how much of the gain is meaningful under this evaluation noise. Evaluation on more open-vocabulary benchmarks such as ALOHa[5] or OpenCHAIR[6] would make the conclusions more convincing.
Citations
[1] Hasanebrahimi A, Huang H, Leckie C, Erfani S. VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs. arXiv. Published August 31, 2026. arXiv:2608.30480.
[2] Che L, Liu TQ, Jia J, Qin W, Tang R, Pavlovic V. Hallucinatory Image Tokens: A Training-free EAZY Approach to Detecting and Mitigating Object Hallucinations in LVLMs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2025:21635-21644.
[3] Gong X, Ming T, Wang X, Wei Z. DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2024:7696-7712. doi:10.18653/v1/2024.emnlp-main.439.
[4] Zhong L, He Z, Zheng J, Li J, Wang ZJ, Kang X. AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM. arXiv. Published March 5, 2026. arXiv:2603.04908.
[5] Petryk S, Chan DM, Kachinthaya A, Zou H, Canny J, Gonzalez JE, Darrell T. ALOHa: A New Measure for Hallucination in Captioning Models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). Association for Computational Linguistics; 2024:342-357. doi:10.18653/v1/2024.naacl-short.30. ACL Anthology
[6] Ben-Kish A, Yanuka M, Alper M, Giryes R, Averbuch-Elor H. Mitigating Open-Vocabulary Caption Hallucinations. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2024:22680-22698. doi:10.18653/v1/2024.emnlp-main.1263.
Questions for authors
Please address my concerns above.
13 comments
Response to Reviewer 8fMm (2/6)
Q2: The novelty and positioning against recent work are unclear. VisER[1] already decomposes object hallucination detection into visual evidence and visual reliance, which is conceptually very close to the proposed VSC/VSR formulation. EAZY[2] and DAMRO[3] are exactly two counter examples for the observation of this paper that "the attention of hallucinated objects is more sparse than real object." Especially, EAZY[2] is also a training-free object hallucination detector and shows that hallucinations can be driven by highly attended image tokens, including cases where attention is concentrated on incorrect but visually similar regions. This suggests that strong or concentrated visual signals do not necessarily imply correct grounding. AdaIAT[3] further shows that stronger attention to previously generated text can reduce hallucination, suggesting that textual contribution should not be directly interpreted as harmful language prior. These works should be discussed in much greater depth, and VisER and EAZY should be included as direct comparisons.
A2. Thank you for highlighting these closely related studies. Existing approaches detect or mitigate object hallucinations by probing object-specific visual logits and textual-prefix support (e.g., VisER), or intervening on visual tokens and decoding behavior (e.g., EAZY, DAMRO, and AdaIAT). However, these approaches provide limited insight into how visual information is internally organized and incorporated into the decoding process. To address these gaps, VSI characterizes visual signal integrity by jointly examining the coherence of attended visual representations and their relative contribution to the decoding process. Specifically, VSC measures the attention-weighted directional agreement among visual representations, while VSR quantifies the relative residual contributions from visual and textual token positions. By integrating these complementary measurements, VSI provides a unified hallucination detection score directly from the original generation, without modifying tokens or requiring regeneration. Moreover, we further emphasize two points.
- VSI does not equate visual attention with correct grounding or textual contributions with harmful priors. EAZY and DAMRO demonstrate that highly attended visual tokens can provide misleading evidence. However, VSC measures representational coherence rather than attention concentration, without assuming that coherence guarantees correct grounding. Similarly, AdaIAT shows that increasing attention to previously generated text can reduce hallucinations. This is consistent with VSR measuring relative source-position contributions rather than treating textual contributions as inherently harmful, since text-position representations may already incorporate visual information.
- VSI achieves superior hallucination detection performance on most evaluated LVLMs. We additionally compare VSI with VisER and EAZY under a unified MSCOCO evaluation protocol. As shown in Table III, VSI outperforms EAZY on all eight LVLMs and VisER on seven of eight models, with improvements of 6.01, 9.42, and 12.28 AUROC points over VisER on InstructBLIP, Qwen2.5-VL-7B, and Qwen3-VL-4B, respectively. Although VisER performs better on LLaVA-1.5-7B, these results support the effectiveness of VSI across diverse model architectures. We have expanded the related-work discussion and incorporated these comparisons into the revised manuscript.
Table III. AUROC (%) comparison of VisER, EAZY, and VSI across eight LVLMs on MSCOCO under a unified evaluation protocol. The best result for each model is shown in bold.
| Model | VisER | EAZY | VSI |
|---|---|---|---|
| LLaVA-1.5-7B | 82.86 | 64.21 | 78.99 |
| InternVL3-8B | 77.23 | 71.21 | 78.33 |
| Shikra | 77.06 | 62.52 | 77.21 |
| InstructBLIP | 77.32 | 63.82 | 83.33 |
| Qwen2-VL-7B | 78.03 | 64.97 | 79.88 |
| Qwen2.5-VL-7B | 68.77 | 65.34 | 78.19 |
| Qwen3-VL-4B | 65.66 | 67.27 | 77.94 |
| Qwen3-VL-8B | 71.48 | 65.34 | 76.68 |
Response to Reviewer 8fMm (3/6)
Q3: The mechanistic claims are stronger than what the paper actually establishes. The paper does not statistically demonstrate that hallucinated generations generally rely on more semantically dispersed visual regions, nor does low VSR uniquely indicate that language priors override visual evidence. VSR itself is still derived from the attention sublayer and measures the visual and textual components of the attention-induced residual update, rather than the full residual stream or the final causal contribution to the output. Therefore, the proposed metrics may serve as diagnostic signals, but the current experiments do not establish the claimed failure mechanisms.
A3. Thanks for your careful assessment of our mechanistic interpretations. We have clarified that VSC and VSR provide complementary diagnostic signals of visual grounding, rather than uniquely identifying two causal failure mechanisms. To substantiate this interpretation, we further emphasize two points: empirical evidence for visual representational coherence and the precise scope and diagnostic value of VSR.
- Grounded object predictions exhibit higher visual representational coherence on average. To directly examine this observation, we have compared the average pairwise cosine similarity of attention-selected visual tokens between grounded and hallucinated object predictions on LLaVA-1.5-7B/MSCOCO. Grounded predictions exhibit higher similarity under both Top-k (0.6193 vs. 0.5771) and Top-p (0.6034 vs. 0.5677) selection (Table I). These results provide direct empirical support for the observed difference in representational coherence, without implying that hallucinations necessarily involve semantically dispersed visual regions.
- VSR provides meaningful diagnostic information without requiring causal attribution. VSR measures the relative magnitudes of attention-induced residual contributions from visual- and text-position tokens at a selected layer. Since text-position representations may already incorporate visual information, a low VSR does not necessarily indicate that language priors override visual evidence, nor does VSR quantify the causal influence of either source on the final prediction. Importantly, its diagnostic effectiveness is supported by the component ablations in Figure 5. VSR alone achieves 77.68% and 74.13% AUROC on LLaVA-1.5-7B and Qwen2.5-VL-7B, respectively, while combining it with VSC further improves performance to 78.99% and 78.19%. These results demonstrate that VSR provides useful and complementary signals for hallucination detection, independently of a causal interpretation.
Response to Reviewer 8fMm (1/6)
A1. Thanks for your insightful comments on the experimental motivation for VSC and VSR. To further strengthen the evidence for both observations, we conduct two additional experiments. First, we directly compare the representational coherence of visual tokens attended by grounded and hallucinated object predictions. Second, we examine how non-assertive textual context involving semantically related objects affects hallucination behavior and visual contributions, without explicitly suggesting an answer. These experiments provide more direct empirical support for the two observations underlying our method. We present the details below.
We have clarified the interpretations of both observations and incorporated these additional experimental findings into the revised manuscript.
Table I. Mean pairwise cosine similarity among attention-selected visual tokens for grounded and hallucinated object mentions on LLaVA-1.5-7B/MSCOCO.
Table II. Effects of non-assertive textual context and image guidance on LLaVA-1.5-7B. Contribution norms and VSR are reported relative to the original condition within each split.