- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
This work proposes a hallucination-mitigation technique for LVLMs called TRACE which corrects visual evidence retrieval at the discovered Visual Retrieval Heads rather than modifying visual processing or answer generation. The causal pathway of object hallucination is traced using path patching, with retrieval queries steered toward the target object region or a null reference based on an external detector's confidence score. The proposed TRACE method is evaluated across POPE, BEAF, and CHAIR. An ablation study on the effect of the targeted heads and steering hyperparameters introduced by TRACE is also conducted.
Critical strengths
-
The paper is well written and easy to follow.
-
The mechanistic analysis of hallucination causes in LVLMs is thorough and convincing. The causal tracing (path patching, attention-pattern analysis) is carefully designed, and each claim is backed by direct evidence, making the conclusions about Visual Retrieval Heads and semantic confusers compelling.
-
The experimental results are strong: TRACE performs well on both object-existence questions and image captioning, achieving substantial hallucination reduction with only a moderate reduction in object recall
-
The resource comparison is valuable for an inference-time method; it addresses my concern that the captioning variant's per-candidate forward passes could incur significant time overhead.
Critical weaknesses
-
TRACE depends on an external detector (YOLO26), which gives rise to two issues: (a) detection errors could propagate into calibrated presence probability, corrupting the ROI and null steering; and (b) a generalization problem—YOLO26's closed vocabulary (~80 classes) cannot cover the object space of open-world object categories, and evaluation is conducted only on COCO, leaving cross-dataset generalization unverified, especially when the detector needs fine-tuning for new domains.
-
The method is model-internal, but only two dated models (LLaVA-1.5, Qwen2-VL) are evaluated; whether the identified heads and corrections transfer to modern architectures such as InternVL 3.5 and Qwen3-VL remains unanswered.
Questions for authors
Do the identified heads and corrections transfer to modern architectures like InternVL 3.5 and Qwen3-VL, despite being evaluated only on dated models?
5 comments
A Friendly Follow-Up on Our Rebuttal Responses
As the rebuttal period is approaching its conclusion on November 18, we would like to sincerely thank you once again for your time and thoughtful feedback. We have carefully considered each of your comments and made every effort to thoroughly address your concerns and questions through detailed responses and additional experiments. We truly hope that these clarifications and new results have helped strengthen your confidence in our work. We would be very grateful if you could share any further thoughts on our responses before the discussion period closes. Should any questions or concerns remain, we would be more than happy to continue the discussion and provide further clarification. We deeply appreciate your valuable time and consideration.
Final Follow-Up on Our Rebuttal Responses
Dear Reviewer,
We sincerely apologize for reaching out once more as the discussion period draws to a close. We would like to respectfully highlight that we have conducted substantial additional experiments specifically addressing your key concerns, including open-vocabulary and cross-dataset generalization, as well as the applicability of TRACE to Qwen3-VL.
We would be truly grateful if you could take a moment to revisit these new findings and consider them in your overall assessment of our work. If any concerns remain, we would greatly appreciate the opportunity to address them.
Thank you once again for your valuable time, thoughtful feedback, and consideration.
Response to Reviewer EUUK — Submission 55074
We sincerely thank you for taking the time to review our paper and for offering constructive feedback. We truly value your effort and have carefully addressed your concerns and questions below.
Weakness 1(a): TRACE depends on an external detector (YOLO26), which gives rise to two issues: (a) detection errors could propagate into calibrated presence probability, corrupting the ROI and null steering.
Response: We really appreciate the reviewer for pointing out this error. Across all 3,000 POPE-COCO adversarial queries, TRACE corrects 73% of LLaVA's incorrect answers (478 cases), while the detector only missed 5.8% (175 cases) causing the model to answer wrongly. Overall, TRACE corrects more errors than it introduces, yielding a net gain of 303 correct answers. Better grounding could further reduce detector-induced errors and improve TRACE's performance. Results with alternative grounding models are presented below.
However, the adverse impact of detector errors on accuracy is undeniable. We acknowledge this limitation and will address it in our future work. Beyond detector-guided correction, our causal analysis provides a foundation for directly repairing the model's retrieval mechanism. Appendix C.3 suggests that similar object-search queries can retrieve evidence from semantic confusers when the target is absent. As an initial proof of concept, finetuning Query Construction Heads' O and V matrices of improves POPE-COCO performance (Appendix C.3.1, Table 13).
Weakness 1(b): YOLO26's closed vocabulary (~80 classes) cannot cover the object space of open-world object categories, and evaluation is conducted only on COCO, leaving cross-dataset generalization unverified, especially when the detector needs fine-tuning for new domains.
Response: Thank you for your question. We are happy to clarify. TRACE is not inherently restricted to YOLO26's vocabulary: it allows the usage of alternative grounding models. To examine open-vocabulary and cross-dataset applicability, we conducted additional experiments with Grounding DINO and SAM3 on POPE-COCO, POPE-AOKVQA, POPE-GQA, AMBER-Existence, and Causal-HalBench. Detailed results and experimental settings are specified in Appendix F.5. The results are summarized below:
These results support TRACE's applicability to open-vocabulary grounding models and datasets beyond COCO. This also shows that TRACE can benefit from stronger grounding signals.
Weakness 2: The method is model-internal, but only two dated models (LLaVA-1.5, Qwen2-VL) are evaluated; whether the identified heads and corrections transfer to modern architectures such as InternVL 3.5 and Qwen3-VL remains unanswered.
Questions For Authors: Do the identified heads and corrections transfer to modern architectures like InternVL 3.5 and Qwen3-VL, despite being evaluated only on dated models?
Response: We thank the reviewer for raising this concern. We have added experiments on Qwen3-VL-8B-Instruct in Appendix B.2.2, including causal circuit analysis and TRACE evaluation on POPE-COCO. The analysis provides evidence for the same three-stage: query construction, visual evidence retrieval, and answer readout. The results are reported in Table 7 and reproduced below.
These results support the generality of the discovered mechanism and the applicability of TRACE to a newer transformer-based VLM, using model-specific heads rather than requiring identical head identities across models.
We sincerely thank you for your thoughtful review. We hope the revised manuscript and the clarifications provided above have addressed your concerns effectively. If our responses have satisfactorily resolved your concerns, we would greatly appreciate it if you could consider updating your review score. However, if you have any remaining concerns or require further clarification, please do not hesitate to let us know.