acceptodds
Withdrawn from ICLR 2027

Reliable Memory Injection into Core Visual Attention Heads for Object Hallucination Mitigation in Large Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) often hallucinate objects that are absent from the input image during open-ended generation. A primary cause is the progressive weakening of visual evidence during autoregressive decoding, which allows language priors to dominate. Existing methods typically enhance visual information at the model or layer level while largely overlooking functional differences among attention heads. To address this limitation, we first analyze visual perception in LVLMs from the perspective of multihead attention and identify Core Visual Attention Heads (CVAHs) through statistics collected across multiple samples. We then introduce Core Visual Attention Head Memory Injection(CAMI), a training-free method. CAMI uses relative visual-attention decay as a state-dependent intervention scheduler, not as a classifier of hallucinated tokens. When triggered, it injects historical visual pattern memory to correct the spatial attention of CVAHs. Experiments on MSCOCO-CHAIR and AMBER-G show that CAMI consistently mitigates object hallucination and outperforms existing state-of-the-art baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Object Hallucination Mitigation, Core Visual Attention Heads, Visual Pattern Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cUSQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates object hallucination in large vision-language models from a multi-head attention perspective. This work first identifies Core Visual Attention Heads (CVAHs), which consistently allocate high attention mass to visual tokens across object mentions. It further observes that faithful object mentions generally receive stronger visual attention than hallucinated ones in shallow layers. Based on these findings, this work propose Core Visual Attention Head Memory Injection (CAMI), a training-free framework that uses relative visual-attention decay for state-dependent intervention scheduling. CAMI constructs historical visual attention memories from non-triggered decoding steps and selectively injects them into CVAHs, while suppressing excessive textual attention in Auxiliary Visual Attention Heads (AVAHs). Experiments on three LVLM backbones demonstrate consistent improvements in hallucination metrics on MSCOCO-CHAIR and AMBER-G. Ablation studies further investigate the contributions of head selection, intervention scheduling, memory injection, and text suppression.

Critical strengths
  1. Consistent performance improvements across multiple LVLM architectures and benchmarks. CAMI achieves substantial reductions in hallucination metrics across three LVLM backbones. Notably, the attention head selections and hyperparameters determined on MSCOCO are directly transferred to AMBER-G without additional tuning, where the method maintains strong performance. These results provide meaningful evidence of CAMI's effectiveness and cross-benchmark transferability.
  2. Well-motivated head-level intervention with convincing ablation studies. The identification of CVAHs through attention statistics across object mentions provides a reasonable motivation for head-specific intervention. The distinction between visual attention mass and spatial attention patterns is also reflected in the method design. Importantly, they compare CAMI against random and periodic intervention policies with approximately matched intervention frequencies, providing evidence for the effectiveness of state-dependent scheduling. Additional ablations show that indiscriminate intervention across all heads or decoding steps performs considerably worse, supporting the importance of selective intervention over generic visual attention enhancement.
Critical weaknesses
  1. Insufficient evidence supporting the reliability of CVAHs and their historical visual attention patterns. CVAHs are identified primarily based on how frequently they allocate high attention mass to visual tokens. However, consistently attending to visual tokens does not necessarily imply attending to the correct image regions or extracting information semantically relevant to the generated objects.
  2. Lack of inference efficiency analysis. CAMI requires continuous attention monitoring, historical memory maintenance, and attention-logit modification across decoder layers. However, the paper does not report inference latency, throughput, or memory overhead, making its practical efficiency difficult to assess.
Questions for authors
  1. Reliability of visual attention memory. Could the authors compare CAMI with controls using unrelated or mismatched historical attention patterns, and examine whether memory injection improves attention alignment with the image regions corresponding to generated objects?
  2. Gating behavior and inference efficiency. Could the authors analyze the characteristics of decoding steps that trigger intervention, evaluate sensitivity to the gating threshold, and report the additional inference latency and memory overhead introduced by CAMI?
  3. Separate contributions of head selection. Table 3 simultaneously changes the heads used for gating and memory injection. Could the authors independently vary these two head selections to clarify their respective contributions?
Reviewer_THeROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

To address hallucination in large vision-language models (LVLMs), where existing methods for enhancing visual information often overlook functional differences among attention heads in multi-head attention, this paper identifies core visual attention heads (CVAHs) through statistical analysis and uses relative visual-attention decay as a gating signal to propose a training-free intervention method for mitigating hallucinations. Based on sample statistics, the authors first define a small subset of attention heads that exhibit high visual attention as CVAHs, arguing that they are sparse, stable, and layer-specific. To mitigate hallucinations, they then propose the CAMI framework, which consists of a gating mechanism based on visual-attention decline, a mechanism for constructing and injecting reliable historical visual patterns, and a text-suppression mechanism. Evaluation is conducted on LLaVA-1.5-7B, Qwen2.5-VL-7B, and InstructBLIP-Vicuna-7B, showing that the proposed modules are effective to some extent. Ablation studies further cover attention-head selection, memory injection mechanisms, text-suppression targets, among others.

Critical strengths

1.The paper offers a new perspective for understanding and mitigating hallucinations in LVLMs by examining attention-head mechanisms. It shifts the focus from visual contribution as a global quantity to a more specific and analyzable component of the model, which can serve as a concrete target for intervention.

2.The proposed method is training-free and requires no additional fine-tuning cost. Its feasibility is validated across multiple LVLM backbones, indicating that the approach is not limited to a single model architecture.

Critical weaknesses

1: Equation (8) uses visual attention mass mlsjm_l^{s_j} as a weighting factor to aggregate historical image logits. This implicitly assumes that a higher attention mass means more reliable logits. However, mlsjm_l^{s_j} only measures how much attention is given to image tokens; it does not show whether that attention is spatially correct. A historical step with high attention mass may still focus mainly on the background or an irrelevant object. Its logits would then receive a high weight and contaminate the memory, which in turn misleads later memory injection. The paper does not provide a mechanism to ensure that the stored patterns are spatially correct, nor does it check whether steps with high attention are actually more reliable. This directly weakens the core claim that CAMI injects “reliable” visual patterns, and it creates a risk that the memory itself becomes a source of hallucination rather than a correction.

2: Section 4.1 defines the gating reference rt−1r_{t-1} using only “non-triggered” historical steps, while the trigger decision gtg_{t} depends entirely on rt−1r_{t-1}. The authors explicitly admit that “reliable” means “consistent with the running reference, not verified factual correctness.” This raises a problem: if the model hallucinates early and its visual attention remains consistently low, then r0r_0 is initialized from a hallucinated state, the subsequent relative decline is near zero, and the gate does not trigger. As a result, the memory injection mechanism is difficult to activate in such cases.

Questions for authors

1.For each backbone, how many attention heads exist per layer? Are the Top-8 selection and the thresholds f>0.5f>0.5 and f<0.1f<0.1 applied identically across all three models? If so, how do you justify the fixed thresholds given that the random baseline 8/H8/H differs across architectures?

2.The second claimed contribution in the Introduction is that the authors reveal that faithful object mentions exhibit stronger shallow-layer visual attention than hallucinated object mentions. The observation that faithful responses have stronger visual attention than hallucinated responses cannot support a standalone contribution, as there is already a related consensus in the literature.

3.In Section 4.1, if the model starts hallucinating from the first generated token, r0r_0 is initialized from that hallucinated state. Can you explain how the gate could recover and trigger intervention in this scenario? What mechanism prevents the reference from being contaminated?

4.Equation (8) uses the historical visual attention mass mlsjm_l^{s_j} as a weight to aggregate historical image logits, implicitly assuming that higher visual attention mass implies more reliable logits. However, mlsjm_l^{s_j} is only a scalar total mass over image tokens and does not reflect whether the attention is spatially correct. A historical step may assign high attention mass to image tokens while focusing on the background or the wrong object. Such logits would receive high weight and contaminate the memory, subsequently misleading memory injection. How can one ensure that historical steps with high mlsjm_l^{s_j} are spatially correct?

5.Several symbols are defined redundantly (e.g., sl,tcores_{l,t}^{core} and mltm_l^t, Eqs. 7 and 12). Could you provide a unified notation table to clarify the intended differences, if any?

Reviewer_95oaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies object hallucination in large vision-language models from the perspective of attention heads. It identifies Core Visual Attention Heads whose visual attention remains relatively stable across object mentions. Based on this observation, the authors propose CAMI, a training-free inference-time intervention method. CAMI monitors visual-attention decay, injects historical visual attention patterns into CVAHs, and suppresses excessive textual attention in auxiliary heads. Experiments on three LVLM backbones and two benchmarks show substantial reductions in CHAIR-based hallucination metrics.

Critical strengths
  1. The paper provides evidence that visual information is concentrated in a sparse and layer-specific set of attention heads, motivating targeted intervention.

  2. CAMI reduces hallucination across three LVLMs and two benchmarks. The matched-rate and component ablations suggest that the improvements come mainly from state-dependent memory injection rather than arbitrary attention modification.

Critical weaknesses
  1. CAMI reduces hallucination but also consistently lowers recall or coverage, raising the concern that it may omit less salient but correct objects and reduce description comprehensiveness.

  2. The paper provides no qualitative success or failure cases, making it difficult to determine whether CAMI improves visual grounding or merely avoids uncertain object mentions.

  3. The evaluation focuses mainly on generative object hallucination. Results on complementary benchmarks such as POPE and MME are needed to assess generalization and possible effects on broader multimodal capabilities.

Questions for authors
  1. Is layer 0 optimal for all three models? Were other layers or multi-layer gating considered?

  2. Since memory uses non-triggered rather than verified states, could noisy or hallucinated patterns be injected?

  3. Please provide success and failure cases explaining both hallucination reduction and recall degradation.

  4. Please provide necessary hyperparameter analyses (e.g., τd\tau_d) and missing experimental details, such as the exact prompts and decoding settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.