acceptodds
Withdrawn from ICLR 2027

State-Dependent Visual Support in Vision-Language Hallucination

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prior studies have associated vision-language model hallucination with apparently conflicting internal visual signals, including reduced visual dependence and highly attended image tokens. We observe a more specific difference between settings. Hallucinated objects receive less visual support in free-form generation but more when the model makes queried object existence decisions. We show that this difference mainly reflects comparisons of internal representations serving different computational roles. To compare these roles directly, we present StateHall, which applies the same measure of visual support to states corresponding to an object introduced during free-form generation, a queried object, and the queried decision. Hallucinated object representations receive less support around intermediate decoder layers in both free-form generation and explicit querying, while relative over-support appears only at the late decision state. A same-image, same-object decomposition shows that the state-role component constitutes most of the final difference between the generated-object and queried-decision gaps, while the component associated with concept origin is substantially smaller. These patterns are also useful for hallucination detection. For free-form hallucination, the resulting midpoint concept-visual support (MCVS) score matches or improves upon recent training-free baselines and transfers across models and datasets. For queried object existence, compact detectors based on the identified states demonstrate strong transfer across benchmarks and complement output confidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, visual grounding, internal representations, hallucination detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1NbNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the seemingly contradictory patterns of visual support exhibited by Vision-Language Models (VLMs) when generating object hallucinations. The authors propose STATEHALL, which measures the degree of visual support for generated objects, queried objects, and query decisions. Furthermore, the study reveals that representations of hallucinated objects receive low visual support in intermediate decoding layers—across both free-generation and object-presence query tasks—yet exhibit disproportionately high visual support during the final query decision stage. Based on these findings, the authors developed a hallucination detector by combining MCVS and STATEHALL. This detector demonstrates competitive performance and strong transferability across several models and benchmarks.

Critical strengths
  1. The findings are somewhat interesting. This paper provides deeper insight into the state-dependent nature of visual support for inference in visual language models (VLMs). In particular, the different patterns of visual support observed between the object representation condition and the decision condition serve to reconcile seemingly contradictory findings from previous research.

  2. The methodology is easy to follow. The proposed system is intuitive and easy to understand. Since this paper clearly describes the three numerical factors and the corresponding decomposition methods, the overall research process is straightforward and clear.

  3. The experiments are relatively comprehensive. This paper investigates the proposed approach on several benchmarks and VLM architectures. It also compares the approach with existing methods and includes statistical analysis.

Critical weaknesses
  1. This paper mainly focuses on the difference between free-form generation and queried object existence. While this comparison is interesting, the research scope is relatively narrow, and the results may be limited. It would be useful to investigate whether similar patterns of condition-dependent patterns emerge in other VLM tasks, such as general VQA, image captioning and multimodal reasoning.

  2. Although this paper evaluates the proposed methodology with different metrics, some aspects of the exploratory analysis can be strengthened. In particular, the COCO test contains only a limited amount of discretionary information (43 for Qwen and 116 for LLaVA), which may limit the robustness of the conclusions. Furthermore, results on the RePOPE are reported only in the combined form. More detailed results (i.e., dividing into three splits) will help to verify whether the proposed method performs consistently for different scenarios and difficulty levels.

  3. While this article discusses several approaches to understanding and detecting hallucinations, it ignores some recent related works [1,2,3,4,5]. A detailed discussion will help to better situate the proposed framework in the most recent literature and provide a comprehensive overview of the field.

[1] Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models, ICML 25.

[2] More Thinking, Less Seeing: Assessing Amplified Hallucination in Multimodal Reasoning Models, NeurIPS 25.

[3] Hallucination-aware Intermediate Representation Edit in Large Vision-Language Models, ICLR 26.

[4] Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models, CVPR 26.

[5] Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding, CVPR 26.

Questions for authors

See weaknesses. If the authors can successfully address my concerns, I would consider raising my ratings.

Reviewer_8Kx5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates contradictory visual-support patterns associated with object hallucinations in VLMs. It observes that hallucinated objects exhibit lower visual support during free-form generation, whereas false-positive object-existence show higher support in late decoder layers. To explain these observations, it introduces STATEHALL, a framework that measures visual support across three states: generated-object, queried-object, and queried-decision. By comparing these states on the same image-object pairs, it decomposes the observed difference into concept-origin and state-role components. Results suggest that hallucinated object representations are under-supported at intermediate layers in both settings, while relative over-support emerges at the late decision state. It further develop training-free and learned hallucination detectors based on these findings, demonstrating promising detection performance and cross-benchmark transfer.

Critical strengths
  1. A systematic characterization of state-dependent hallucinations. The paper identifies an interesting discrepancy between free-form generation and object-existence querying and investigates it through a unified framework. Introducing the queried-object state enables more controlled comparisons, while the same-image/object analysis provides quantitative evidence that hallucination features depend on the measured computational state.
  2. Promising detection performance and transferability. The proposed MCVS method is simple, training-free, and achieves competitive results across models and datasets. The queried detectors are also evaluated under direct benchmark transfer without target-set recalibration. This protocol provides useful evidence that the identified state features retain predictive value under distribution shift.
Critical weaknesses
  1. Insufficient isolation of the computational-role effect. The proposed decomposition does not fully disentangle computational role from token-position and contextual differences. Specifically, the queried-object state represents the object tokens, while the queried-decision state is extracted from the final prompt position, which may encode substantially different semantic and contextual information.
  2. Limited generalizability and robustness. The complete three-state decomposition is validated only on Qwen2.5-VL, leaving its generalizability across architectures uncertain. Additionally, the learned detectors are trained on a single RePOPE split, without evaluating stability across different training partitions.
Questions for authors
  1. How to isolate the effect of computational role from token-position and contextual differences? Could controlled experiments with alternative query formulations or token positions verify that the reported 77% contribution primarily reflects computational role?
  2. Does the complete three-state decomposition generalize to other VLM architectures? Could the authors also evaluate learned-detector stability across different training splits and provide a more comprehensive accuracy-efficiency comparison to justify the additional computational overhead?
Reviewer_mHsMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the seemingly contradictory visual support patterns associated with object hallucinations in free-form generation and object-existence queries. To explain these differences, the authors propose STATEHALL, a state-aware framework that measures visual support across generated-object, queried-object, and queried-decision states, and decomposes the observed differences into concept-origin and state-role components. Experiments show that hallucinated objects receive weaker visual support at intermediate layers, while false-positive decisions exhibit stronger support at late layers, with state-role differences accounting for most of the discrepancy. These findings further enable hallucination detectors that demonstrate competitive performance and cross-benchmark transferability.

Critical strengths
  1. The paper identifies seemingly contradictory visual support patterns between free-form generation and object-existence querying. By distinguishing object representations from decision states, it provides a meaningful explanation for these observations.
  2. Although STATEHALL-U and STATEHALL-L underperform the output-confidence baseline, STATEHALL-HYBRID achieves only a modest absolute AUROC improvement of 0.007 on Causal-HalBench. The practical benefits of extracting internal visual support features remain unclear given their additional computational overhead.
  3. The paper fixes K = 8 and averages the top-K cosine similarities without sufficiently justifying these design choices. Ablations over different K values and alternative aggregation functions, such as maximum or median, would help establish the robustness of the reported patterns.
  4. The proposed MCVS and STATEHALL detectors translate the identified internal patterns into compact hallucination detection scores. Experiments demonstrate competitive free-form detection performance and promising cross-benchmark transferability, with state features providing complementary information to output confidence.
Critical weaknesses
  1. Although the paper evaluates visual support patterns across multiple models, the complete same-image, same-object decomposition is established only on Qwen2.5-VL-3B. Reproducing the state-role dominance on additional architectures would provide stronger evidence for the generality of the main finding.
  2. The state-based detectors underperform simple output-confidence baselines, while STATEHALL-HYBRID provides only a modest AUROC improvement of 0.007 on Causal-HalBench. The practical benefit of extracting internal visual support features therefore remains uncertain relative to their additional computational cost.
  3. The paper fixes (K=8) and averages the Top-K cosine similarities without sufficiently justifying these design choices. An ablation over different K values and aggregation strategies, including the variance of selected similarities, would help establish the robustness of the reported state-dependent hallucination patterns.
Questions for authors
  1. The complete same-image, same-object decomposition is conducted only on Qwen2.5-VL-3B. Does the dominance of the state-role component also hold across other model architectures, such as LLaVA?
  2. STATEHALL-U and STATEHALL-L underperform the output-confidence baseline, while STATEHALL-HYBRID achieves only a modest AUROC improvement of 0.007 on Causal-HalBench. How do the authors justify the practical value of these internal visual support features given their additional computational overhead?
  3. The visual support measure uses a fixed (K=8) and averages the Top-K cosine similarities. How sensitive are the main findings to different K values and aggregation strategies, such as maximum, median, or variance-aware measures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.