acceptodds
Under review as a conference paper at ICLR 2027

Visual Signal Integrity for Detecting Object Hallucinations in LVLMs

18 views

Abstract

Large vision-language models (LVLMs) demonstrate remarkable content generation capabilities but are prone to object hallucinations, i.e., generating contextually plausible yet visually ungrounded descriptions. This imposes critical reliability risks in multimodal understanding. Detecting such hallucinations remains challenging because they often arise from breakdowns in visual information preservation that are not captured by existing methods, which inspect only isolated points along the visual-to-language pathway. In this paper, we reveal that grounded object tokens exhibit higher semantic cohesion among attended visual tokens and stronger visual contributions in the residual stream than hallucinated ones. Inspired by these observations, we propose a training-free diagnostic framework, Visual Signal Integrity (VSI), to quantify the integrity of visual grounding during generation. Specifically, VSI introduces two complementary statistics: Visual Semantic Cohesion (VSC), which measures the geometric concentration of attended visual tokens to assess whether coherent visual evidence is formed, and Visual Steering Ratio (VSR), which measures the relative dominance of visual over textual contributions in the residual stream. By combining these two statistics, VSI provides a unified score for detecting breakdowns in visual information flow. Extensive experiments on multiple hallucination benchmarks and diverse LVLMs show that VSI delivers a reliable and interpretable solution for hallucination detection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Hallucination Detection, LVLM

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_8fMmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VSI, a training-free method for detecting hallucinations in LVLMs. It uses VSC to measure how coherent the attended visual tokens are, and VSR to measure the relative visual vs. textual contribution during decoding. The two signals are combined to score whether a generated object is visually grounded. The method is evaluated across multiple LVLM architectures and hallucination benchmarks.

Critical strengths
  1. The paper provides a simple and interpretable way to separate visual representation quality from visual influence during decoding.

  2. The method is training-free and can be applied across different LVLM architectures without additional model training.

  3. The experiments cover multiple models and hallucination settings, with ablations showing that VSC and VSR provide complementary signals.

  4. The paper is generally easy to follow.

Critical weaknesses

This is a well-made paper. However, I still have strong concerns regarding the novelty, soundness and evaluation quality of this work.

  1. The motivation behind both observations is not sufficiently convincing. For VSC, comparing object-related visual tokens with randomly sampled tokens is expected to produce higher similarity, since tokens belonging to the same object already share appearance, texture, and spatial context. This does not demonstrate that hallucinated objects should have less coherent visual representations. For VSR, the POPE intervention mainly introduces an explicit instruction bias by telling the model that an object exists or does not exist. This is different from the language priors usually discussed in hallucination work, such as object co-occurrence or attribute priors, so the experiment does not strongly support the claimed language-prior interpretation.

  2. The novelty and positioning against recent work are unclear. VisER[1] already decomposes object hallucination detection into visual evidence and visual reliance, which is conceptually very close to the proposed VSC/VSR formulation. EAZY[2] and DAMRO[3] are exactly two counter examples for the observation of this paper that "the attention of hallucinated objects is more sparse than real object." Especially, EAZY[2] is also a training-free object hallucination detector and shows that hallucinations can be driven by highly attended image tokens, including cases where attention is concentrated on incorrect but visually similar regions. This suggests that strong or concentrated visual signals do not necessarily imply correct grounding. AdaIAT[3] further shows that stronger attention to previously generated text can reduce hallucination, suggesting that textual contribution should not be directly interpreted as harmful language prior. These works should be discussed in much greater depth, and VisER and EAZY should be included as direct comparisons.

  3. The mechanistic claims are stronger than what the paper actually establishes. The paper does not statistically demonstrate that hallucinated generations generally rely on more semantically dispersed visual regions, nor does low VSR uniquely indicate that language priors override visual evidence. VSR itself is still derived from the attention sublayer and measures the visual and textual components of the attention-induced residual update, rather than the full residual stream or the final causal contribution to the output. Therefore, the proposed metrics may serve as diagnostic signals, but the current experiments do not establish the claimed failure mechanisms.

  4. The distinction from prior attention- and representation-based methods is somewhat overstated. Both VSC and VSR are constructed from attention weights and hidden representations, and VSR can be viewed as a more detailed attention-output contribution measure rather than a fundamentally different source of information. Stronger ablations against simpler attention- and representation-based signals are needed to show what is specifically gained by the proposed formulations.

  5. The theoretical analysis is limited. The propositions are mainly algebraic reformulations: VSC rewrites a weighted pairwise similarity as a squared norm, while VSR decomposes the attention output according to source tokens. These identities may be useful computationally, but they do not constitute substantial theoretical contributions by themselves.

  6. The evaluation has important limitations. Both MSCOCO and Objects365 rely on fixed object taxonomies and CHAIR-style matching, which can introduce noisy labels for open-ended generations due to missing categories, coarse ontology mappings, and incomplete semantic coverage. Objects365 expands the vocabulary but does not remove the underlying issue. Given that some of the improvements over baselines are relatively small, it is difficult to determine how much of the gain is meaningful under this evaluation noise. Evaluation on more open-vocabulary benchmarks such as ALOHa[5] or OpenCHAIR[6] would make the conclusions more convincing.

Citations

[1] Hasanebrahimi A, Huang H, Leckie C, Erfani S. VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs. arXiv. Published August 31, 2026. arXiv:2608.30480.

[2] Che L, Liu TQ, Jia J, Qin W, Tang R, Pavlovic V. Hallucinatory Image Tokens: A Training-free EAZY Approach to Detecting and Mitigating Object Hallucinations in LVLMs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2025:21635-21644.

[3] Gong X, Ming T, Wang X, Wei Z. DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2024:7696-7712. doi:10.18653/v1/2024.emnlp-main.439.

[4] Zhong L, He Z, Zheng J, Li J, Wang ZJ, Kang X. AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM. arXiv. Published March 5, 2026. arXiv:2603.04908.

[5] Petryk S, Chan DM, Kachinthaya A, Zou H, Canny J, Gonzalez JE, Darrell T. ALOHa: A New Measure for Hallucination in Captioning Models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). Association for Computational Linguistics; 2024:342-357. doi:10.18653/v1/2024.naacl-short.30. ACL Anthology

[6] Ben-Kish A, Yanuka M, Alper M, Giryes R, Averbuch-Elor H. Mitigating Open-Vocabulary Caption Hallucinations. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2024:22680-22698. doi:10.18653/v1/2024.emnlp-main.1263.

Questions for authors

Please address my concerns above.

13 comments
AuthorsOpenReview ↗

Response to Reviewer 8fMm (1/6)

Q1: The motivation behind both observations is not sufficiently convincing. For VSC, comparing object-related visual tokens with randomly sampled tokens is expected to produce higher similarity, since tokens belonging to the same object already share appearance, texture, and spatial context. This does not demonstrate that hallucinated objects should have less coherent visual representations. For VSR, the POPE intervention mainly introduces an explicit instruction bias by telling the model that an object exists or does not exist. This is different from the language priors usually discussed in hallucination work, such as object co-occurrence or attribute priors, so the experiment does not strongly support the claimed language-prior interpretation.

A1. Thanks for your insightful comments on the experimental motivation for VSC and VSR. To further strengthen the evidence for both observations, we conduct two additional experiments. First, we directly compare the representational coherence of visual tokens attended by grounded and hallucinated object predictions. Second, we examine how non-assertive textual context involving semantically related objects affects hallucination behavior and visual contributions, without explicitly suggesting an answer. These experiments provide more direct empirical support for the two observations underlying our method. We present the details below.

  • Grounded object predictions exhibit higher visual representation coherence than hallucinated predictions. While Figure 2 compares object-related visual tokens with randomly sampled tokens, we further examine the visual representations attended by actual generated object mentions. Specifically, we select visual tokens based on the generated object token's attention using Top-k and Top-p strategies, and compute their average pairwise cosine similarity. As shown in Table I, grounded predictions consistently exhibit higher visual representation coherence than hallucinated predictions under both selection strategies. These results provide direct empirical evidence for the relationship between attended visual representation coherence and hallucination, beyond the original object-versus-random comparison.
  • Even non-assertive semantic context increases hallucinations while weakening visual contributions. To distinguish semantic contextual bias from explicit answer induction, we further conduct controlled experiments using textual context that mentions semantically related objects without stating whether the queried object exists (Appendix D.4). We evaluate 500 ground-truth negative examples from each of the POPE Popular and Adversarial subsets, where incorrect “Yes” responses correspond to object hallucinations. As shown in Table II, misleading context increases the hallucination rate from 35% to 57% on Popular and from 63% to 76% on Adversarial, accompanied by reduced visual contributions and relative VSR values of 0.48× and 0.54×, respectively. Conversely, image-guided prompts reduce hallucination rates to 22% and 44%, while substantially increasing visual contributions and VSR. These contrasting results demonstrate that semantically associated textual context can influence hallucination behavior and relative visual contributions without explicitly prescribing an answer, providing stronger experimental support for the interpretation of VSR beyond direct instruction bias.

We have clarified the interpretations of both observations and incorporated these additional experimental findings into the revised manuscript.

Table I. Mean pairwise cosine similarity among attention-selected visual tokens for grounded and hallucinated object mentions on LLaVA-1.5-7B/MSCOCO.

Visual-token selectionMean Grounded similarity vs. Mean Hallucinated similarity
Top-k (k=20k=20)0.6193 vs. 0.5771
Top-p (p=0.5p=0.5)0.6034 vs. 0.5677

Table II. Effects of non-assertive textual context and image guidance on LLaVA-1.5-7B. Contribution norms and VSR are reported relative to the original condition within each split.

SplitConditionHallucination Rate ↓Relative Visual ContributionRelative Textual ContributionRelative VSR
PopularOriginal35%1.00×1.00×1.00×
Image-guided22%1.88×1.01×1.84×
Misleading prior57%0.54×1.11×0.48×
AdversarialOriginal63%1.00×1.00×1.00×
Image-guided44%1.84×0.99×1.86×
Misleading prior76%0.58×1.08×0.54×
AuthorsOpenReview ↗

Response to Reviewer 8fMm (2/6)

Q2: The novelty and positioning against recent work are unclear. VisER[1] already decomposes object hallucination detection into visual evidence and visual reliance, which is conceptually very close to the proposed VSC/VSR formulation. EAZY[2] and DAMRO[3] are exactly two counter examples for the observation of this paper that "the attention of hallucinated objects is more sparse than real object." Especially, EAZY[2] is also a training-free object hallucination detector and shows that hallucinations can be driven by highly attended image tokens, including cases where attention is concentrated on incorrect but visually similar regions. This suggests that strong or concentrated visual signals do not necessarily imply correct grounding. AdaIAT[3] further shows that stronger attention to previously generated text can reduce hallucination, suggesting that textual contribution should not be directly interpreted as harmful language prior. These works should be discussed in much greater depth, and VisER and EAZY should be included as direct comparisons.

A2. Thank you for highlighting these closely related studies. Existing approaches detect or mitigate object hallucinations by probing object-specific visual logits and textual-prefix support (e.g., VisER), or intervening on visual tokens and decoding behavior (e.g., EAZY, DAMRO, and AdaIAT). However, these approaches provide limited insight into how visual information is internally organized and incorporated into the decoding process. To address these gaps, VSI characterizes visual signal integrity by jointly examining the coherence of attended visual representations and their relative contribution to the decoding process. Specifically, VSC measures the attention-weighted directional agreement among visual representations, while VSR quantifies the relative residual contributions from visual and textual token positions. By integrating these complementary measurements, VSI provides a unified hallucination detection score directly from the original generation, without modifying tokens or requiring regeneration. Moreover, we further emphasize two points.

  • VSI does not equate visual attention with correct grounding or textual contributions with harmful priors. EAZY and DAMRO demonstrate that highly attended visual tokens can provide misleading evidence. However, VSC measures representational coherence rather than attention concentration, without assuming that coherence guarantees correct grounding. Similarly, AdaIAT shows that increasing attention to previously generated text can reduce hallucinations. This is consistent with VSR measuring relative source-position contributions rather than treating textual contributions as inherently harmful, since text-position representations may already incorporate visual information.
  • VSI achieves superior hallucination detection performance on most evaluated LVLMs. We additionally compare VSI with VisER and EAZY under a unified MSCOCO evaluation protocol. As shown in Table III, VSI outperforms EAZY on all eight LVLMs and VisER on seven of eight models, with improvements of 6.01, 9.42, and 12.28 AUROC points over VisER on InstructBLIP, Qwen2.5-VL-7B, and Qwen3-VL-4B, respectively. Although VisER performs better on LLaVA-1.5-7B, these results support the effectiveness of VSI across diverse model architectures. We have expanded the related-work discussion and incorporated these comparisons into the revised manuscript.

Table III. AUROC (%) comparison of VisER, EAZY, and VSI across eight LVLMs on MSCOCO under a unified evaluation protocol. The best result for each model is shown in bold.

ModelVisEREAZYVSI
LLaVA-1.5-7B82.8664.2178.99
InternVL3-8B77.2371.2178.33
Shikra77.0662.5277.21
InstructBLIP77.3263.8283.33
Qwen2-VL-7B78.0364.9779.88
Qwen2.5-VL-7B68.7765.3478.19
Qwen3-VL-4B65.6667.2777.94
Qwen3-VL-8B71.4865.3476.68
AuthorsOpenReview ↗

Response to Reviewer 8fMm (3/6)

Q3: The mechanistic claims are stronger than what the paper actually establishes. The paper does not statistically demonstrate that hallucinated generations generally rely on more semantically dispersed visual regions, nor does low VSR uniquely indicate that language priors override visual evidence. VSR itself is still derived from the attention sublayer and measures the visual and textual components of the attention-induced residual update, rather than the full residual stream or the final causal contribution to the output. Therefore, the proposed metrics may serve as diagnostic signals, but the current experiments do not establish the claimed failure mechanisms.

A3. Thanks for your careful assessment of our mechanistic interpretations. We have clarified that VSC and VSR provide complementary diagnostic signals of visual grounding, rather than uniquely identifying two causal failure mechanisms. To substantiate this interpretation, we further emphasize two points: empirical evidence for visual representational coherence and the precise scope and diagnostic value of VSR.

  • Grounded object predictions exhibit higher visual representational coherence on average. To directly examine this observation, we have compared the average pairwise cosine similarity of attention-selected visual tokens between grounded and hallucinated object predictions on LLaVA-1.5-7B/MSCOCO. Grounded predictions exhibit higher similarity under both Top-k (0.6193 vs. 0.5771) and Top-p (0.6034 vs. 0.5677) selection (Table I). These results provide direct empirical support for the observed difference in representational coherence, without implying that hallucinations necessarily involve semantically dispersed visual regions.
  • VSR provides meaningful diagnostic information without requiring causal attribution. VSR measures the relative magnitudes of attention-induced residual contributions from visual- and text-position tokens at a selected layer. Since text-position representations may already incorporate visual information, a low VSR does not necessarily indicate that language priors override visual evidence, nor does VSR quantify the causal influence of either source on the final prediction. Importantly, its diagnostic effectiveness is supported by the component ablations in Figure 5. VSR alone achieves 77.68% and 74.13% AUROC on LLaVA-1.5-7B and Qwen2.5-VL-7B, respectively, while combining it with VSC further improves performance to 78.99% and 78.19%. These results demonstrate that VSR provides useful and complementary signals for hallucination detection, independently of a causal interpretation.
Reviewer_aJEjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a training-free Hallucination Detection methods for Large Vision Language Models based on analyzing visual information flow during generation. Specifically, the authors argued that grounded object tokens exhibit higher semantic cohesion than hallucinated ones, which tend to scatter in divergent directions (Observation 1), and hallucinated tokens tend to have more influence from textual tokens than visual ones in the residual stream (Observation 2). To this end, this work proposed two Attention-based scoring metrics, Visual Semantic Cohesion (VSC) and Visual Steering Ratio (VSR) to tackle these two observations, respectively. VSC derives attention-weighted cosine similarities among visual tokens, which measures whether the attended tokens are highly cluttered.VSR decomposes the ratio of visual and textual attention in the residual stream. These two metrics are combined to form the final method, Visual Signal Integrity (VSI). VSI achieves competitive hallucination detection performance across multiple benchmarks and LVLM's model families.

Critical strengths

Extensive and significant experimental results: The paper tested their method across multiple LVLM's backbones with diverse sizes and multiple well-established hallucination benchmarks. The results are impressive, with VSI achieving state-of-the-art results consistently.

Practical and Light-weight method: VSI requires zero training data with minimal compute overhead, while achieving impressive detection results. The method was also proven to generalize well across architectures.

Critical weaknesses
  • W1: The motivational experiments do not seem to support the observations strongly: For Observation 1, the authors claimed that grounded objects have more semantic cohesion in the representation space than hallucinated ones, and they proved this statement through comparing the pairwise cosine similarities of tokens within a ground truth object with randomly sampled tokens in the image. I believe this result does not back up the authors' claims, since patches within one object are expected to be more similar than random patches simply because of spatial contiguity and shared local appearance and visual features. A randomly sampled set of visual tokens is not shown to represent hallucinatory objects, and the experiment involves neither hallucinated mentions nor the attention weights on which VSC is defined.
    Meanwhile, Observation 2 claims that language priors suppress visual evidence and cause hallucination in the residual stream. They supported this by adding "Yes" and "No" language priors to the input prompt. However, only the percentage of Yes answers are reported, which does not correspond to hallucination rate, while the behavior is never compared between true and hallucinated answers. The experiment only shows that textual contribution increases a marginal amount (9-12%) when a strong language prior is intentionally injected into the input prompt, which is a predictable occurrence. This however does not correspond to actual hallucination. Also it's worth mentioning that the textual contribution is already substantially larger than visual contribution in the neutral prompt (around 3.2 times), so text dominance alone doesn't seem to be a strong signal for hallucination.

  • W2 Incremental Contributions compared to SVAR: Both components seem to build heavily on SVAR's attention mass analysis in visual tokens. VSC added the directional coherence of the attended visual tokens in the formula, however a high VSC still requires high SVAR-type attention to the image so SVAR's attention mass contribution in VSC is still substantial. In Table 10, the top-p variant where only unweighted direction similarities was used ended up performing worse than SVAR alone, which shows that VSC is still highly dependent on SVAR's attention weights to perform well.
    Meanwile, VSR is highly similar to SVAR in terms of both motivation and computation: hallucinated tokens relies less on visual tokens than grounded ones, and is computed similarly. VSR uses the same attention weights as SVAR, but multiplies each by the corresponding token's projected value vector before summing, then compares the size of the resulting visual sum with that of the textual sum. The novelty therefore lies only in how this signal is measured, and the paper does not show that the new measurement captures anything SVAR missest. Gains of the individual components over SVAR are also inconsistent across models. On LLaVA both exceed SVAR, but on Qwen2.5-VL-7B neither does (VSC 75.76, VSR 74.13 vs. SVAR 76.16); only their combination VSI outperforms it (78.19).

Questions for authors
  • Q1: How does higher similarity between tokens from the same objects compared with those of random tokens (Observation 1) support the claim that hallucinated objects have diffused semantic directions while grounded objects are concentrated? Neither hallucinated mentions nor attended tokens appear in this experiment. How do pairwise random tokens similarities reflect the dispersion of hallucinated tokens?
  • Q2: What's the difference in textual and visual contribution in the residual space in true and hallucinated tokens in more natural settings (image captioning, VQA...)? Do they hold the same magnitude range and change patterns compared with the results in Figure 3, which is more explicit in terms of injected language prior. Q3: In Figure 8, the value range of VSR seems to differ significantly. While LLaVA's is only around 0.3, Instruct BLIP's VSR is around 1.9 . If text dominance is the norm for some models even with neutral prompt, at what point does it indicate hallucination? Also the absolute change in value of VSR when injected with language prior seems to be quite small too, decreasing only around 0.07 for InstructBLIP. Is there a model-agnostic interpretation of VSR's scale?
  • Q4: How do VSR and detection performance vary with token position? As the text context grows during generation, more text tokens compete with the same set of visual tokens for a fixed attention budget, so VSR may become lower as the context progresses. Is VSR performance still preserved for tokens in longer context (think hundreds or thousands of tokens or more), which is more susceptible to hallucination.
  • Q5: VSR relies on the assumption that we can decompose the residual stream into textual and visual contributions cleanly, and the contribution from text tokens are purely textual. But previous works [1], [2] have established that textual tokens already carry relayed visual information after self-attention on visual tokens. So the text contribution vector in Equation might have confounding visual contributions in them. How much does Δh_txt change when the image is replaced by a blank image while the text is held fixed, and does VSR's interpretation as visual vs. textual dominance still hold?
  • Q6: What is SVAR's AUROC when computed at the same single layer l used for VSI? Since VSI's layer is tuned per model while SVAR uses its original layer range, this comparison would show whether the gains come from coherence and value-weighting or from layer selection.
    [1] Zhang, Zhi, et al. "Cross-modal information flow in multimodal large language models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
    [2] Kaduri, Omri, Shai Bagon, and Tali Dekel. "What’s in the Imageƒ A Deep-Dive into the Vision of Vision Language Models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
20 comments
AuthorsOpenReview ↗

Response to Reviewer aJEj (1/7)

Q1: The motivational experiments do not seem to support the observations strongly: For Observation 1, the authors claimed that grounded objects have more semantic cohesion in the representation space than hallucinated ones, and they proved this statement through comparing the pairwise cosine similarities of tokens within a ground truth object with randomly sampled tokens in the image. I believe this result does not back up the authors' claims, since patches within one object are expected to be more similar than random patches simply because of spatial contiguity and shared local appearance and visual features. A randomly sampled set of visual tokens is not shown to represent hallucinatory objects, and the experiment involves neither hallucinated mentions nor the attention weights on which VSC is defined. How does higher similarity between tokens from the same objects compared with those of random tokens (Observation 1) support the claim that hallucinated objects have diffused semantic directions while grounded objects are concentrated? Neither hallucinated mentions nor attended tokens appear in this experiment. How do pairwise random tokens similarities reflect the dispersion of hallucinated tokens?

A1. Thanks for your constructive comments on the motivation experiments. The original experiment in Figure 2 was intended to provide an initial observation of visual representation cohesion, rather than directly characterize hallucinated objects. To further examine its connection to hallucination, we have extended the analysis to attention-selected visual tokens associated with grounded and hallucinated object predictions. The results show consistently higher average cohesion for grounded objects under both Top-k and Top-p selection, while these attention-guided coherence measures also demonstrate useful hallucination detection performance. Based on these findings, we have refined Observation 1 to more accurately reflect the experimental evidence and its motivation for VSC.

  • Clarifications of Figure 2's Original Purpose. Figure 2 was intended to examine whether object-relevant visual tokens exhibit more coherent representations than randomly sampled visual tokens, motivating the use of cosine similarity to measure representational cohesion. Random tokens serve as a reference rather than a proxy for hallucinated objects.
  • Direct Comparison Between Grounded and Hallucinated Objects. Building on the Top-k/Top-p analysis in Appendix D.5, we further compare the visual representations attended by grounded and hallucinated object tokens on LLaVA-1.5-7B/MSCOCO. For each generated object token, we use its attention weights to select visual tokens through Top-k (k=20k=20) or Top-p (p=0.5p=0.5), and compute their unweighted mean pairwise cosine similarity. As shown in Table I, visual tokens attended by grounded object predictions exhibit higher average similarity under both strategies (0.6193 vs. 0.5771 for Top-k; 0.6034 vs. 0.5677 for Top-p).
  • Attention-Guided Visual Cohesion Provides Discriminative Signals for Hallucination Detection. The Top-k and Top-p coherence measures achieve 71.25% and 72.98% AUROC, respectively, showing that the directional coherence of attention-selected visual tokens provides useful information for hallucination detection. The complete VSC further combines visual attention magnitude with directional coherence, achieving 77.86% AUROC. Based on the original and additional experiments, we have revised Observation 1 as follows: Visual tokens within annotated object regions exhibit higher representational cohesion than randomly sampled visual tokens, while grounded object predictions tend to attend to more cohesive visual representations than hallucinated predictions.

Table I. Mean pairwise cosine similarity and hallucination detection performance of attention-selected visual tokens for grounded and hallucinated object predictions on LLaVA-1.5-7B/MSCOCO at layer index 10.

Visual-token selectionMean Grounded similarity vs. Mean Hallucinated similarityAUROC (%) ↑
Top-k (k=20k=20)0.6193 vs. 0.577171.25
Top-p (p=0.5p=0.5)0.6034 vs. 0.567772.98
AuthorsOpenReview ↗

Response to Reviewer aJEj (2/7)

Q2: Observation 2 claims that language priors suppress visual evidence and cause hallucination in the residual stream. They supported this by adding "Yes" and "No" language priors to the input prompt. However, only the percentage of Yes answers are reported, which does not correspond to hallucination rate, while the behavior is never compared between true and hallucinated answers. The experiment only shows that textual contribution increases a marginal amount (9-12%) when a strong language prior is intentionally injected into the input prompt, which is a predictable occurrence. This however does not correspond to actual hallucination. Also it's worth mentioning that the textual contribution is already substantially larger than visual contribution in the neutral prompt (around 3.2 times), so text dominance alone doesn't seem to be a strong signal for hallucination.

A2. Our experiments provide empirical evidence linking weakened visual contributions to increased object hallucinations. Figure 3 examines how induced language priors affect visual and textual contributions during decoding; Appendix D.4 further connects these changes to actual hallucination errors through controlled textual interventions. Importantly, the results show that visual contributions decrease substantially more than textual contributions increase under misleading contexts, indicating that the weakened visual influence cannot be explained by increased textual contributions alone.

  • Clarifications of Figure 3's Purpose. Figure 3 is designed to examine changes in modality-specific contributions under controlled language induction, rather than directly measuring hallucination rates. Under Yes-Induce and No-Induce prompts, the visual-to-textual contribution ratio decreases from 0.32 to 0.24 and 0.23, respectively (approximately 25% and 28% reductions). These results show that both types of language induction weaken the relative contribution of visual information, despite shifting answer preferences in opposite directions.
  • Direct Evidence Linking Visual Steering to Hallucinations. Appendix D.4 (Table 12, reproduced as Table II below) provides direct evidence connecting visual steering with actual object hallucinations through controlled textual interventions.
    • Experimental setting. We evaluate LLaVA-1.5-7B on 500 ground-truth negative examples from each of the POPE Popular and Adversarial subsets, where incorrect "Yes" responses constitute object hallucinations. We compare image-guided instructions with misleading textual contexts that introduce semantic biases without explicitly specifying the answer.
    • Empirical evidence. On Popular, misleading context increases the hallucination rate from 35% to 57%, accompanied by a 52% reduction in relative VSR. Conversely, image guidance reduces the hallucination rate to 22%, while increasing relative VSR by 84%. Similar trends hold on Adversarial, where misleading context increases the hallucination rate from 63% to 76% (VSR: 0.54×), whereas image guidance reduces it to 44% (VSR: 1.86×). These opposing trends show that changes in relative visual steering are consistently associated with actual hallucination rates, beyond the answer-preference shifts examined in Figure 3.
  • Visual Contributions Decrease Substantially More Than Textual Contributions Increase under Misleading Context. In Figure 3, textual contributions increase by only 9–12%, but visual contributions simultaneously decrease by 18–20%. More importantly, Table II shows that misleading context reduces visual contributions by 46% and 42% on Popular and Adversarial, respectively, whereas textual contributions increase by only 11% and 8%. Conversely, image-guided context increases visual contributions by 88% and 84%, while textual contributions remain nearly unchanged (1.01× and 0.99×). These results indicate that increased hallucinations are associated with weakened visual contributions relative to textual contributions, rather than textual dominance alone.

We have included the above discussions in the revised version.

Table II. Effects of textual interventions on object hallucination rates and visual/textual contributions on POPE using LLaVA-1.5-7B.

SplitConditionHallucination Rate ↓Relative Visual ContributionRelative Textual ContributionRelative VSR
PopularOriginal35%1.00×1.00×1.00×
Image-guided22%1.88×1.01×1.84×
Misleading prior57%0.54×1.11×0.48×
AdversarialOriginal63%1.00×1.00×1.00×
Image-guided44%1.84×0.99×1.86×
Misleading prior76%0.58×1.08×0.54×
AuthorsOpenReview ↗

Response to Reviewer aJEj (3/7)

Q3: In Figure 8, the value range of VSR seems to differ significantly. While LLaVA's is only around 0.3, Instruct BLIP's VSR is around 1.9 . If text dominance is the norm for some models even with neutral prompt, at what point does it indicate hallucination? Also the absolute change in value of VSR when injected with language prior seems to be quite small too, decreasing only around 0.07 for InstructBLIP. Is there a model-agnostic interpretation of VSR's scale?

A3. VSR measures the relative magnitude of visual and textual contributions to the residual stream, rather than directly indicating whether an object prediction is grounded or hallucinated. Although its absolute values vary across architectures, its interpretation as a measure of relative visual contribution remains consistent. Importantly, VSR is not intended to provide a universal numerical threshold for hallucination detection. We clarify these distinctions below.

  • Textual Dominance Does Not Necessarily Indicate Hallucination. VSR measures the relative magnitudes of visual and textual contributions to the residual stream, but does not directly indicate whether an object prediction is grounded or hallucinated. For example, a VSR of 0.3 in LLaVA indicates a smaller visual contribution relative to the textual contribution, but does not imply that the generated object is hallucinated. A smaller VSR indicates weaker relative visual contributions, which may be associated with hallucination but is not sufficient to establish it.
  • Model-Dependent VSR Scales. Different LVLM architectures employ different visual representations, projection mechanisms, and residual update scales, leading to different VSR ranges. Thus, the values of 0.3 for LLaVA and 1.9 for InstructBLIP should not be directly compared as indicators of grounding quality. VSR provides a consistent measure of relative modality contributions, while its ability to distinguish grounded from hallucinated predictions is evaluated within each model, rather than relying on a universal threshold.
  • Small VSR Changes and Detection Performance. Figure 8 examines how language priors affect relative visual contributions, rather than directly measuring hallucination detection performance. Although InstructBLIP exhibits a relatively small average VSR decrease of approximately 0.07 under language induction, VSR alone achieves 81.23% AUROC in distinguishing grounded from hallucinated object predictions. This shows that a small intervention-induced change in average VSR does not necessarily imply weak hallucination detection performance, as the latter depends on the score distributions of grounded and hallucinated predictions.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Response to Reviewer Ld39 (1/4)

W1: The paper does not reconcile its GLSim results with the published ones, and the gap changes the main comparison. On MSCOCO, the GLSim paper reports 83.7/94.2 AUROC/AUPR for LLaVA-1.5-7B and 83.0/94.9 for Shikra. Table 1 reports 80.97/90.14 and 68.91/81.68. The difference is larger still for InstructBLIP: 85.0 AUROC in the original paper versus 56.97 here. These gaps are far beyond the standard deviations in Appendix D.1. More importantly, they reverse the ranking. VSI scores 77.21 on Shikra and 83.33 on InstructBLIP, which beats the reproduced GLSim numbers but not the published ones. Objects365 shows the same pattern: the published GLSim values are 72.6/74.6 for LLaVA and 69.7/75.9 for Shikra, compared with 68.40/71.83 and 62.70/68.50 in this submission.

Q1: Baseline reproduction. Before applying the unified protocol, can the authors reproduce the published GLSim results on the overlapping model–dataset pairs? A controlled explanation of the Shikra drop from 83.0 to 68.91 AUROC and the InstructBLIP drop from 85.0 to 56.97 would materially affect my assessment.

A1. Thanks for your careful examination of the GLSim comparison and for highlighting the importance of evaluation consistency. We would like to clarify that our evaluation follows the published GLSim methodology and builds upon its official implementation, with model-specific adaptations where necessary. Importantly, GLSim, VSI, and all other baselines are evaluated on the same generated object predictions and annotations, ensuring a consistent comparison across methods. We further examine GLSim's layer sensitivity and find that VSI maintains higher AUROC on both Shikra and InstructBLIP even when GLSim's layer pairs are selected to maximize test performance.

  • Our evaluation of GLSim follows the published methodology under a unified protocol. The publicly released GLSim evaluation code directly supports LLaVA-1.5-7B/13B. For other architectures, including Shikra and InstructBLIP, we adapt its scoring procedure following the original paper's methodology and reported hyperparameter settings. All methods are evaluated on identical generated responses and object annotations to ensure a fair comparison. Although both studies use 5,000 sampled MSCOCO images, their evaluated object prediction sets differ. For example, the original GLSim paper reports 15,724 grounded and 4,727 hallucinated objects for Shikra, whereas our evaluation contains 10,414 grounded and 5,091 hallucinated objects. Therefore, the published scores and our reproduced results are not directly comparable as evaluations on identical test predictions.
  • GLSim remains below VSI even when its layer pairs are optimized on the test set. For Shikra, we use the reported text/image layer pair (30, 27). For InstructBLIP, where an explicit layer pair is not provided, we select the pair using validation AUROC. We further examine layer sensitivity by scanning candidate layer pairs and identifying the highest AUROC on our test set. As shown in Table I, GLSim improves from 68.91% to 76.61% on Shikra and from 56.97% to 71.95% on InstructBLIP, while VSI achieves 77.21% and 83.33%, respectively. These results demonstrate that although layer selection affects GLSim performance, VSI maintains higher AUROC on both models. We report these test-optimized results as an additional diagnostic comparison favorable to GLSim, separately from our standard validation-selected evaluation.
  • Our performance comparisons are based on identical evaluation conditions. Directly substituting the published GLSim scores into our comparison tables would change the rankings on some models, but would also mix results obtained from different evaluation settings and generated object predictions. Therefore, we evaluate GLSim and VSI on the same object predictions and annotations to ensure a fair comparison. We have further clarified the scope of these comparisons in the revised manuscript.

Table I. Comparison of GLSim and VSI on MSCOCO. GLSim layer pairs are selected to maximize test AUROC for diagnostic purposes, whereas VSI uses validation-selected configurations. AUROC is reported in %.

ModelTest-selected text layerTest-selected image layerGLSim best test AUROC (%)VSI AUROC (%)
Shikra313276.6177.21
InstructBLIP122371.9583.33
AuthorsOpenReview ↗

Response to Reviewer Ld39 (2/4)

Q2: The scores predict hallucinations more convincingly than they identify two distinct failure mechanisms. Proposition 2 separates source-position terms within one attention update. It does not separate causal visual information from language priors: text-position states may already contain image information, and taking norms removes direction and downstream effects. VSC also depends on both total visual attention and directional agreement, as Appendix D.5 acknowledges. A low VSC therefore does not by itself identify failed concept formation, and a low VSR does not show that a language prior overrode an intact visual concept.

A2. Thanks for your thoughtful comments on the interpretation of VSI and its components. We agree that measuring hallucination-related internal signals and identifying their underlying causal mechanisms are distinct objectives. Our approach aims to provide complementary diagnostic measurements of visual grounding, rather than uniquely identify two separate causal failure mechanisms. We clarify the diagnostic scope of VSI, the interpretation of its components, and the supporting experimental evidence below.

  • Clarifications of the Diagnostic Scope of VSI. VSI examines two complementary properties of visual grounding: the coherence of attended visual representations and the relative residual contributions from visual and textual token positions during decoding. These properties provide different perspectives on the internal signals associated with a generated prediction, rather than corresponding to two mutually exclusive failure mechanisms. In particular, the individual VSC and VSR scores allow us to examine these properties separately, while their combination provides a unified hallucination detection score. Thus, the diagnostic value of VSI lies in characterizing multiple aspects of visual grounding, rather than uniquely locating the causal origin of each hallucination.
  • Clarifications of the Diagnostic Interpretations of VSC and VSR. VSC intentionally combines attention strength and representational coherence to characterize coherent visual support for a generated prediction. We agree that a low VSC does not necessarily imply failed visual concept formation, as it may reflect weak visual attention, low representational coherence, or both. For VSR, Proposition 2 decomposes the attention-induced residual update by source-token positions, rather than causally separating visual information from language priors. Since text-position states may already contain visual information, and contribution norms do not establish their causal effects on the final prediction, We have therefore clarified VSR as a measure of relative source-position contributions during decoding, rather than a direct indicator of language-prior override.
  • Experimental Evidence Supports the Diagnostic Value of VSI. Figure 5 shows that VSC alone provides effective discrimination between grounded and hallucinated object predictions. For VSR, Figure 3 demonstrates that controlled changes in textual context are accompanied by systematic shifts in both decoding behavior and residual contributions, while Figure 5 further confirms its individual detection capability. Importantly, combining VSC and VSR achieves higher AUROC than either component alone on both LLaVA-1.5-7B and Qwen2.5-VL-7B, supporting their complementary diagnostic value.

We have revised the corresponding descriptions in the manuscript to clarify the diagnostic scope of VSI and distinguish its measurable properties from causal explanations of hallucination.

AuthorsOpenReview ↗

Response to Reviewer Ld39 (3/4)

Q3: The POPE result lacks a clear response-level scoring and labeling protocol. Appendix C.4 says that the queried object is scored, while C.6 compares the detector output with that object's ground-truth existence. It is therefore unclear whether Table 3 measures an incorrect generated answer or simply the absence of the queried object.

A3. Thanks for pointing out the ambiguity in our description of the POPE evaluation protocol. We clarify that Table 4 evaluates the correctness of generated Yes/No responses, rather than the presence or absence of the queried object. Specifically, for each image-question pair, we perform a single forward pass to predict the Yes/No answer and compute VSI using the attention weights and hidden states obtained in that pass. Each response is labeled correct if the predicted answer matches the ground-truth answer, and incorrect otherwise. Thus, a correct “No” is labeled correct, while both an incorrect “Yes” (false positive) and an incorrect “No” (false negative) are labeled incorrect. AUROC and AUPR are computed to evaluate how well VSI distinguishes correct from incorrect responses, rather than detecting object absence. We have clarified this response-level scoring and labeling procedure in Appendices C.4 and C.6 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.