- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates object hallucination in large vision-language models from a multi-head attention perspective. This work first identifies Core Visual Attention Heads (CVAHs), which consistently allocate high attention mass to visual tokens across object mentions. It further observes that faithful object mentions generally receive stronger visual attention than hallucinated ones in shallow layers. Based on these findings, this work propose Core Visual Attention Head Memory Injection (CAMI), a training-free framework that uses relative visual-attention decay for state-dependent intervention scheduling. CAMI constructs historical visual attention memories from non-triggered decoding steps and selectively injects them into CVAHs, while suppressing excessive textual attention in Auxiliary Visual Attention Heads (AVAHs). Experiments on three LVLM backbones demonstrate consistent improvements in hallucination metrics on MSCOCO-CHAIR and AMBER-G. Ablation studies further investigate the contributions of head selection, intervention scheduling, memory injection, and text suppression.
Critical strengths
- Consistent performance improvements across multiple LVLM architectures and benchmarks. CAMI achieves substantial reductions in hallucination metrics across three LVLM backbones. Notably, the attention head selections and hyperparameters determined on MSCOCO are directly transferred to AMBER-G without additional tuning, where the method maintains strong performance. These results provide meaningful evidence of CAMI's effectiveness and cross-benchmark transferability.
- Well-motivated head-level intervention with convincing ablation studies. The identification of CVAHs through attention statistics across object mentions provides a reasonable motivation for head-specific intervention. The distinction between visual attention mass and spatial attention patterns is also reflected in the method design. Importantly, they compare CAMI against random and periodic intervention policies with approximately matched intervention frequencies, providing evidence for the effectiveness of state-dependent scheduling. Additional ablations show that indiscriminate intervention across all heads or decoding steps performs considerably worse, supporting the importance of selective intervention over generic visual attention enhancement.
Critical weaknesses
- Insufficient evidence supporting the reliability of CVAHs and their historical visual attention patterns. CVAHs are identified primarily based on how frequently they allocate high attention mass to visual tokens. However, consistently attending to visual tokens does not necessarily imply attending to the correct image regions or extracting information semantically relevant to the generated objects.
- Lack of inference efficiency analysis. CAMI requires continuous attention monitoring, historical memory maintenance, and attention-logit modification across decoder layers. However, the paper does not report inference latency, throughput, or memory overhead, making its practical efficiency difficult to assess.
Questions for authors
- Reliability of visual attention memory. Could the authors compare CAMI with controls using unrelated or mismatched historical attention patterns, and examine whether memory injection improves attention alignment with the image regions corresponding to generated objects?
- Gating behavior and inference efficiency. Could the authors analyze the characteristics of decoding steps that trigger intervention, evaluate sensitivity to the gating threshold, and report the additional inference latency and memory overhead introduced by CAMI?
- Separate contributions of head selection. Table 3 simultaneously changes the heads used for gating and memory injection. Could the authors independently vary these two head selections to clarify their respective contributions?