- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how LVLMs utilize multiple context sources (image, prompt, and generation history) during generation to mitigate object hallucinations. The authors identify three key observations: (1) visual reliance peaks when generating object-related content in the early stage, (2) adaptive visual steering is preferable to fixed amplification, and (3) textual contexts provide complementary corrective signals. Based on these findings, they propose AIMS, a training-free method that adaptively steers attention queries using visual, prefilled-text, and generated-text contexts to mitigate hallucinations. Experiments across multiple models and benchmarks demonstrate its effectiveness in reducing hallucinations.
Critical strengths
-
The paper provides empirical evidence on how visual attention evolves during generation and how attention intervention can be designed more effectively for hallucination mitigation, yielding three useful observations. These findings naturally motivate a practical, training-free solution for mitigating object hallucinations in LVLMs.
-
The proposed AIMS is evaluated comprehensively across representative models, decoding strategies, and benchmarks, with results consistently supporting its effectiveness.
Critical weaknesses
-
While the paper presents useful findings that motivate the proposed method, these findings are validated only on Qwen2.5-VL-3B, raising concerns about their reliability.
-
The paper lacks ablations comparing adaptive single- and multi-context steering. Since prefilled texts (P) may also contain misleading information, such as POPE queries [R1] that explicitly mention absent objects, and generated text (G) may already contain hallucinations, it remains unclear whether incorporating P and G consistently improves over Adaptive-V.
[R1] Li Y, Du Y, Zhou K, et al. Evaluating object hallucination in large vision-language models. EMNLP, 2023.
Questions for authors
Please refer to the Critical Weaknesses section.
3 comments
I appreciate the authors' detailed and timely response.
W1: Validation on a different VLM family (e.g., InternVL) would be ideal for demonstrating generality. That said, the additional Qwen3.5-9B results are acceptable.
W2: My concern has been addressed. I recommend incorporating the relevant discussion into the manuscript to better support the proposed method's effectiveness.
Overall, I will keep my positive score (3).
We sincerely thank the reviewer for the positive assessment and constructive feedback. We will incorporate the additional experimental results and relevant discussions into the revised manuscript. We also plan to further extend our implementation framework to support a broader range of LVLM architectures and facilitate cross-family evaluation in future work.
We appreciate the reviewer's thoughtful suggestions, which have helped us strengthen our work.
We sincerely thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the value of our empirical findings and proposed method. We address each concern below.
W1: Generalizability of the empirical observations.
We thank the reviewer for this concern. To validate the generalizability of our observations, we conducted additional experiments on Qwen3.5-9B using the same 500 images. We divided generated sequences into five length-based bins with approximately equal sample counts and observed consistent visual attention trends across decoding steps, similar to Qwen2.5-VL-3B. These results further support the generalizability of our observations across LVLMs. We will include the corresponding curves in the revised manuscript.
W2: Potential risks of incorporating P/G.
We thank the reviewer for this concern. Although P/G may contain misleading information, they also provide useful context for instruction understanding and coherent generation. AIMS therefore adaptively weights V/P/G rather than uniformly enhancing textual information.
To investigate potential hallucination propagation through generated context, we analyzed 500 CHAIR responses generated by Qwen2.5-VL-3B under greedy decoding, including 222 containing hallucinations.
Here, Nhall and Nobj denote hallucinated and total object mentions after the first hallucination, respectively. Nnewmention counts hallucinated mentions belonging to categories other than the first hallucinated category, while Nnew counts distinct new hallucinated categories. The second metric is computed over 209 descriptions with subsequent object mentions, while the other metrics are computed over all 222 hallucinated descriptions.
Although the first hallucination occurs relatively early (36.8% of generation), the average hallucination rate among subsequent object mentions is only 15.4%. Moreover, of the 1.20 subsequent hallucinated mentions per description, only 0.41 involve new categories, corresponding to 0.27 distinct new categories. These results suggest that subsequent hallucinations mainly repeat existing errors rather than continuously introduce new object categories.
Moreover, Figure 5 directly compares adaptive single- and multi-context steering, showing that full AIMS achieves lower hallucination rates and higher F1 than V-only. Together, these results support the benefits of adaptively incorporating P/G despite their potential imperfections.