acceptodds
Under review as a conference paper at ICLR 2027

Attend to Vision When It Matters: Visual Dependency-Aware Attention Rebalancing for Large Vision-Language Models

Abstract

Large Vision-Language Models (LVLMs) commonly suffer from visual hallucinations, generating content that deviates from visual evidence. This problem stems from the dominance of linguistic priors in autoregressive generation and the insufficient visual grounding. Existing inference-time attention interventions mitigate hallucinations by strengthening visual attention, but typically rely on native attention patterns and adopt static or uniform enhancement strategies, failing to distinguish informative visual tokens, identify layers that benefit most from intervention, or adapt to the dynamic demand for visual evidence during generation. Such indiscriminate intervention may alleviate hallucinations at the cost of language generation quality. To address this, we propose ViDRA, a fine-grained attention rebalancing framework. During prefill, ViDRA estimates visual token importance by jointly considering representation stability and instruction relevance, while deriving layer-wise intervention strength from visual activity across Transformer layers. During decoding, a lightweight Visual Dependency Gate predicts the visual dependency of the upcoming token, enabling visual attention to be enhanced only when needed. Extensive experiments on visual hallucination benchmarks and general vision-language tasks show that ViDRA effectively reduces visual hallucinations while preserving language generation quality and inference efficiency, achieving state-of-the-art performance across multiple LVLM backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Visual Hallucination, Attention Rebalancing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8acqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ViDRA, a visual dependency-aware attention rebalancing framework for mitigating hallucinations in Large Vision-Language Models (LVLMs). The method estimates visual token importance based on low-frequency representation stability and instruction relevance, and derives layer-wise intervention strengths from visual activity during prefill. During decoding, a lightweight Visual Dependency Gate predicts the visual dependency of upcoming tokens to dynamically adjust visual attention enhancement. Experiments across three LVLM backbones on hallucination and general vision-language benchmarks demonstrate improved hallucination mitigation while largely preserving general task performance and inference efficiency.

Critical strengths
  1. Well-motivated and fine-grained attention intervention. The paper identifies the limitations of static visual attention enhancement and proposes a framework that jointly considers visual token importance, layer-wise activity, and step-wise visual dependency. The analysis in Fig. 3 provides useful evidence that visual dependency varies across generation steps and is not reliably reflected by native attention weights.

  2. Strong empirical performance across multiple LVLMs. The method achieves promising results on three LVLM backbones across hallucination and general vision-language benchmarks. The component ablations and efficiency analysis further demonstrate the practical benefits of the proposed framework, particularly its ability to mitigate hallucinations without substantially degrading general task performance.

Critical weaknesses
  1. Insufficient validation of the learned Visual Dependency Gate. Although the ablation study demonstrates that the Gate contributes substantially to performance improvements, it remains unclear whether these gains arise from accurate token-wise dependency prediction or simply from reducing excessive visual attention enhancement. Comparisons with simpler gating strategies under matched intervention strengths are needed to establish the specific contribution of the learned Gate.

  2. Lack of direct evaluation of language generation quality. A central motivation of ViDRA is to mitigate hallucinations while preserving the model's intrinsic language generation capability. However, the general benchmarks in Table 2 primarily evaluate visual question-answering accuracy rather than language fluency, coherence, or repetition. Although qualitative examples are provided, quantitative evaluations of generation quality are needed to substantiate this central claim.

Questions for authors
  1. Regarding the contribution of the Visual Dependency Gate. Could the authors compare the learned Gate with simpler alternatives, such as constant gating or token-category-based gating, under matched visual attention enhancement strengths? This would help clarify whether the improvements primarily arise from accurate visual dependency prediction or from reducing unnecessary attention intervention.

  2. Regarding the preservation of language generation quality. A central motivation of ViDRA is to mitigate hallucinations while preserving language generation quality. However, the general benchmarks in Table 2 mainly evaluate visual question-answering accuracy rather than language fluency or coherence. Could the authors provide direct quantitative evaluations of language generation quality, such as fluency, repetition, and informativeness, on open-ended generation tasks? This would help substantiate the claimed trade-off between hallucination mitigation and language generation quality.

  3. Regarding visual dependency supervision. The dependency labels are constructed from the log-probability differences between full-vision and vision-drop conditions, with negative values truncated to zero. How sensitive is the method to these design choices? Could the authors evaluate alternative label construction strategies to further validate the reliability of the proposed supervision signal?

  4. Regarding cross-backbone generalization. Are separate Visual Dependency Gates trained for LLaVA-1.5-7B, Qwen3-VL-8B, and InternVL3.5-8B? If so, could the authors clarify the corresponding training configurations and costs? How are differences in hidden-state dimensions and feature distributions handled across these models?

  5. Regarding baseline comparison fairness. In Tables 1 and 2, several attention intervention baselines, including VisFlow and PAI, underperform Greedy decoding on certain benchmarks. Could the authors clarify whether these methods were appropriately adapted to different LVLM architectures? Would tuning their intervention strengths under the same validation protocol affect the reported performance gaps?

Reviewer_mVZCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the limitations of static and uniform visual-attention interventions for mitigating hallucinations in LVLMs. It proposes ViDRA, which estimates visual-token importance using a discrete Fourier transform-based foreground prior and trains a lightweight gate to predict the visual dependency of each upcoming token. Together with layer-wise visual activity, these signals enable adaptive intervention across visual tokens, Transformer layers, and decoding steps. Experiments across multiple LVLMs and benchmarks demonstrate the effectiveness of ViDRA.

Critical strengths
  • ViDRA provides a well-motivated extension of existing attention interventions by automatically adjusting the enhancement across visual tokens, layers, and generation steps.
  • The method is extensively evaluated across multiple LVLMs and hallucination benchmarks, with comprehensive ablation and efficiency analyses supporting its effectiveness.
Critical weaknesses
  • The proposed Visual Dependency Gating approach is conceptually not novel. It primarily trains a lightweight gate to replace the existing visual dependency estimation (Eq. 3) obtained via extra forward (e.g., VCD and PAI), thereby shifting the computation from inference to offline training to reduce latency. Moreover, adapting the method to a new LVLM requires reconstructing a large gate training dataset, raising practical concerns.

  • The weak correlation between native visual attention and visual dependency is a key observation motivating ViDRA, yet essential experimental details, including the evaluated LVLM, dataset, and attention aggregation strategy, are not provided. Besides, this observation is not validated across multiple representative LVLMs, raising concerns about its reproducibility and reliability.

Questions for authors
  1. Given the observation that native visual attention fails to reflect visual dependency, why does ViDRA still rely on native attention distributions to estimate instruction relevance RiR_i and layer-wise visual activity ωl\omega_l?
  2. Why introduce a new low-pass reconstruction error for estimating the foreground prior instead of directly applying the channel-wise stability score from LaSt-ViT? The two formulations appear to rely on the same low-pass reconstruction signal and differ mainly in normalization and aggregation. Please clarify their methodological distinction and the empirical benefit of the proposed formulation.
  3. Additional references on Attention Intervention to consider:

[1] Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. Jiang Z, Chen J, Zhu B, et al. CVPR 2025.

Reviewer_Sow5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the visual hallucination problem in LVLMs and points out that existing attention-based visual token selection methods can be affected by linguistic biases, making it difficult to accurately identify important visual tokens. Based on two observations, the authors propose ViDRA: first, foreground regions of visual tokens exhibit more stable low-frequency features, which can be used to distinguish foreground and background tokens; second, the authors measure the visual dependency of generated tokens by comparing the prediction differences with and without visual information. These two signals provide information about important visual tokens and the visual dependency of each generated token, which enables dynamic adjustment of visual attention in the LLM. Experiments demonstrate that ViDRA can reduce visual hallucinations while preserving generation quality.

Critical strengths
  1. The paper provides an interesting analysis of visual token importance and token-level visual dependency in LVLMs, and proposes a unified framework to dynamically adjust visual attention.

  2. The method avoids fine-tuning the LVLM backbone and introduces a lightweight gate, making it applicable to existing LVLMs.

Critical weaknesses

Tables 1 and 2 compare ViDRA with training-free baselines, while ViDRA requires additional supervised training, making the comparison potentially less fair. Moreover, Table 3 shows that removing the trained Dependency Gate leads to a substantial performance drop, suggesting that the attention rebalancing mechanism alone may not be sufficient to achieve competitive performance. Therefore, it remains unclear whether the gains come from the proposed attention rebalancing mechanism or the additional training introduced by the Gate.

Questions for authors

1.The paper assumes that foreground visual tokens are more informative, but this assumption is not directly validated. An oracle experiment using ground-truth foreground masks to guide visual token importance should help quantify the upper bound of foreground-based selection.

2.For Tables 1 and 2, the authors should perform at least three independent runs for ViDRA and recent competing methods, and report the mean and variance to verify the reliability of the performance improvements.

3.The ablation study should evaluate the contribution of each component to the final performance.

4.Since visual dependency labels require additional full-vision and vision-drop inference, the paper should report the cost of offline label construction and quantify the inference efficiency gain introduced by the Gate.

5.The paper categorizes generated tokens into semantic categories (Entity, Attribute, Action, Relation, and Syntax), but does not provide the token-level category annotations or the classification procedure. In addition, the paper lacks the training details of the Visual Dependency Gate for Qwen3-VL and InternVL3.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.