acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Causal Tracing and Editing Visual Evidence Retrieval to Mitigate Vision-Language Model Hallucinations

Abstract

Large Vision-Language Models (LVLMs) have demonstrated strong capabilities but remain prone to object hallucinations. Existing training-free methods focus on modifying visual processing or answer generation, rather than causally identifying the sources of hallucinations. We are the first to causally trace the information flow underlying hallucinations and discover Visual Retrieval Heads: attention heads that ground visual evidence at queried-object positions to causally influence predictions. We also discover that evidence retrieved by these heads significantly contributes to the model's confidence. When this evidence comes from a similar but incorrect region, it can make a false object strongly supported, leading to an overconfident hallucination. Yet, existing non-causal methods minimally intervene in this retrieval process. As such, they cannot correct overconfident hallucinations. Based on this insight, we propose TRACE, a causal intervention method that corrects how visual evidence is gathered by the identified heads. Experiments on object-existence prediction and image captioning benchmarks show that TRACE significantly outperforms prior methods by up to 11.7%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Hallucination, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EUUKOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This work proposes a hallucination-mitigation technique for LVLMs called TRACE which corrects visual evidence retrieval at the discovered Visual Retrieval Heads rather than modifying visual processing or answer generation. The causal pathway of object hallucination is traced using path patching, with retrieval queries steered toward the target object region or a null reference based on an external detector's confidence score. The proposed TRACE method is evaluated across POPE, BEAF, and CHAIR. An ablation study on the effect of the targeted heads and steering hyperparameters introduced by TRACE is also conducted.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The mechanistic analysis of hallucination causes in LVLMs is thorough and convincing. The causal tracing (path patching, attention-pattern analysis) is carefully designed, and each claim is backed by direct evidence, making the conclusions about Visual Retrieval Heads and semantic confusers compelling.

  3. The experimental results are strong: TRACE performs well on both object-existence questions and image captioning, achieving substantial hallucination reduction with only a moderate reduction in object recall

  4. The resource comparison is valuable for an inference-time method; it addresses my concern that the captioning variant's per-candidate forward passes could incur significant time overhead.

Critical weaknesses
  1. TRACE depends on an external detector (YOLO26), which gives rise to two issues: (a) detection errors could propagate into calibrated presence probability, corrupting the ROI and null steering; and (b) a generalization problem—YOLO26's closed vocabulary (~80 classes) cannot cover the object space of open-world object categories, and evaluation is conducted only on COCO, leaving cross-dataset generalization unverified, especially when the detector needs fine-tuning for new domains.

  2. The method is model-internal, but only two dated models (LLaVA-1.5, Qwen2-VL) are evaluated; whether the identified heads and corrections transfer to modern architectures such as InternVL 3.5 and Qwen3-VL remains unanswered.

Questions for authors

Do the identified heads and corrections transfer to modern architectures like InternVL 3.5 and Qwen3-VL, despite being evaluated only on dated models?

5 comments
AuthorsOpenReview ↗

Response to Reviewer EUUK — Submission 55074

We sincerely thank you for taking the time to review our paper and for offering constructive feedback. We truly value your effort and have carefully addressed your concerns and questions below.

Weakness 1(a): TRACE depends on an external detector (YOLO26), which gives rise to two issues: (a) detection errors could propagate into calibrated presence probability, corrupting the ROI and null steering.

Response: We really appreciate the reviewer for pointing out this error. Across all 3,000 POPE-COCO adversarial queries, TRACE corrects 73% of LLaVA's incorrect answers (478 cases), while the detector only missed 5.8% (175 cases) causing the model to answer wrongly. Overall, TRACE corrects more errors than it introduces, yielding a net gain of 303 correct answers. Better grounding could further reduce detector-induced errors and improve TRACE's performance. Results with alternative grounding models are presented below.

However, the adverse impact of detector errors on accuracy is undeniable. We acknowledge this limitation and will address it in our future work. Beyond detector-guided correction, our causal analysis provides a foundation for directly repairing the model's retrieval mechanism. Appendix C.3 suggests that similar object-search queries can retrieve evidence from semantic confusers when the target is absent. As an initial proof of concept, finetuning Query Construction Heads' O and V matrices of improves POPE-COCO performance (Appendix C.3.1, Table 13).

Weakness 1(b): YOLO26's closed vocabulary (~80 classes) cannot cover the object space of open-world object categories, and evaluation is conducted only on COCO, leaving cross-dataset generalization unverified, especially when the detector needs fine-tuning for new domains.

Response: Thank you for your question. We are happy to clarify. TRACE is not inherently restricted to YOLO26's vocabulary: it allows the usage of alternative grounding models. To examine open-vocabulary and cross-dataset applicability, we conducted additional experiments with Grounding DINO and SAM3 on POPE-COCO, POPE-AOKVQA, POPE-GQA, AMBER-Existence, and Causal-HalBench. Detailed results and experimental settings are specified in Appendix F.5. The results are summarized below:

MethodPOPE Adversarial (avg.) AccPOPE Adversarial (avg.) F1AMBER-Existence AccCausal-HalBench AccCausal-HalBench F1
Greedy70.676.470.978.678.1
TRACE + YOLO2679.278.485.884.779.9
TRACE + Grounding DINO81.182.290.785.883.3
TRACE + SAM382.983.990.687.684.5

These results support TRACE's applicability to open-vocabulary grounding models and datasets beyond COCO. This also shows that TRACE can benefit from stronger grounding signals.

Weakness 2: The method is model-internal, but only two dated models (LLaVA-1.5, Qwen2-VL) are evaluated; whether the identified heads and corrections transfer to modern architectures such as InternVL 3.5 and Qwen3-VL remains unanswered.

Questions For Authors: Do the identified heads and corrections transfer to modern architectures like InternVL 3.5 and Qwen3-VL, despite being evaluated only on dated models?

Response: We thank the reviewer for raising this concern. We have added experiments on Qwen3-VL-8B-Instruct in Appendix B.2.2, including causal circuit analysis and TRACE evaluation on POPE-COCO. The analysis provides evidence for the same three-stage: query construction, visual evidence retrieval, and answer readout. The results are reported in Table 7 and reproduced below.

POPE-COCO splitGreedy AccuracyGreedy F1TRACE AccuracyTRACE F1
Random92.191.590.589.6
Popular88.788.489.488.5
Adversarial86.686.588.787.8
Overall89.188.889.588.6

These results support the generality of the discovered mechanism and the applicability of TRACE to a newer transformer-based VLM, using model-specific heads rather than requiring identical head identities across models.


We sincerely thank you for your thoughtful review. We hope the revised manuscript and the clarifications provided above have addressed your concerns effectively. If our responses have satisfactorily resolved your concerns, we would greatly appreciate it if you could consider updating your review score. However, if you have any remaining concerns or require further clarification, please do not hesitate to let us know.

AuthorsOpenReview ↗

A Friendly Follow-Up on Our Rebuttal Responses

As the rebuttal period is approaching its conclusion on November 18, we would like to sincerely thank you once again for your time and thoughtful feedback. We have carefully considered each of your comments and made every effort to thoroughly address your concerns and questions through detailed responses and additional experiments. We truly hope that these clarifications and new results have helped strengthen your confidence in our work. We would be very grateful if you could share any further thoughts on our responses before the discussion period closes. Should any questions or concerns remain, we would be more than happy to continue the discussion and provide further clarification. We deeply appreciate your valuable time and consideration.

AuthorsOpenReview ↗

Final Follow-Up on Our Rebuttal Responses

Dear Reviewer,

We sincerely apologize for reaching out once more as the discussion period draws to a close. We would like to respectfully highlight that we have conducted substantial additional experiments specifically addressing your key concerns, including open-vocabulary and cross-dataset generalization, as well as the applicability of TRACE to Qwen3-VL.

We would be truly grateful if you could take a moment to revisit these new findings and consider them in your overall assessment of our work. If any concerns remain, we would greatly appreciate the opportunity to address them.

Thank you once again for your valuable time, thoughtful feedback, and consideration.

Reviewer_MJZeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates object hallucinations in LVLMs through causal tracing, identifying a circuit involving query construction, visual evidence retrieval, and answer readout. Based on this analysis, the paper proposes TRACE, which uses an external object detector to steer selected retrieval heads toward detected regions or a null reference. Experiments on LLaVA-1.5-7B and Qwen2-VL-7B show improvements on POPE, BEAF, and CHAIR.

Critical strengths
  • Detailed mechanistic analysis. The paper uses path patching and controlled interventions to connect visual retrieval with downstream predictions, which is sound and detailed. The distinction between retrieving incorrect evidence and simply paying insufficient attention to images provides a useful account of confident hallucinations.
  • Consistent improvements within the evaluated setting. TRACE reduces object hallucinations across both binary existence questions and free-form captioning on two VLMs, providing practical support for the proposed intervention.
Critical weaknesses
  • Limited novelty of the visual retrieval head discovery. The discovery of visual retrieval head seems to be a critical contribution of this work. However, as stated in Bi et al., Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025, arXiv:2412.18108), and Park et al., Retrieval Heads Meet Vision (arXiv:2608.27417), already provided many experiments and analysis. The hallucination-specific circuit analysis indeed hold its unique value, but the paper does not clearly establish what is new about the head discovery itself, so the contribution, though valuable, may be limited.
  • The contribution beyond external detection is not isolated. At inference time, TRACE receives object-presence confidence and bounding boxes from an external detector. For object-existence questions, this information already closely matches what the model must predict. Without detector-only and detector-informed prompting baselines using the same external information, it is difficult to attribute the gains specifically to causal head intervention.
Questions for authors
  • How does TRACE ensuress preserving general multimodal understanding? Object recall decreases on both models in the captioning evaluation, making it important to establish whether reduced hallucination comes at the cost of useful visual content. Results on general multimodal benchmarks would directly affect this assessment.
  • Does the identified three-stage circuit persist in newer VLMs and more challenging hallucination settings? The current evidence from LLaVA-1.5-7B and Qwen2-VL-7B on POPE, BEAF, and CHAIR leaves its relevance to current architectures, training recipes, and tasks uncertain. Also, the benchmarks used in this article are relatively outdated, more evaluatoins on frontier benchmarks would address this concern to a great extent
  • It seems that Object-existence QA and captioning use separately discovered heads and different intervention procedures: direct answer intervention versus candidate verification and logit adjustment. How would TRACE handle general or mixed multimodal queries involving descriptions, attributes, counting, and relations? Is there a unified inference procedure, or is its applicability currently restricted to the two evaluated task formats?
8 comments
AuthorsOpenReview ↗

Response to Reviewer MJZe — Submission 55074 (Part 1)

Thank you for reviewing our paper and for offering constructive feedback. We truly value your effort and have addressed your concerns and questions below.

Weakness 1: Limited novelty of the visual retrieval head discovery. The discovery of visual retrieval head seems to be a critical contribution of this work. However, as stated in Bi et al., Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025, arXiv:2412.18108), and Park et al., Retrieval Heads Meet Vision (arXiv:2608.27417), already provided many experiments and analysis. The hallucination-specific circuit analysis indeed hold its unique value, but the paper does not clearly establish what is new about the head discovery itself, so the contribution, though valuable, may be limited.

Response: Thanks for highlighting these related works. We understand why their findings on specialized visual heads raise this concern about the novelty of our discovery. Bi et al. [1] identify visual-processing heads through attention statistics and examine their importance through masking. Park et al. [2] identify Visual Retrieval Heads using attention to ground-truth image regions and validate their role in visual grounding through causal ablation. Both provide valuable findings about components involved in visual processing.

The technical gap lies in their reliance on observed attention patterns for discovery, followed by causal interventions to confirm the selected heads' importance. Head-level ablation alone cannot reconstruct the information flow: which upstream components construct the retrieval signal, how retrieved information reaches downstream components and contributes to a decision. Park et al. explicitly acknowledge this gap in Section 6 of [2], stating that their attention-based analysis captures where evidence is routed rather than what information is read or transformed, and identify activation patching and causal mediation as future directions.

In contrast, our investigation is motivated by understanding this information flow, with causal interventions guiding circuit discovery throughout. Through path patching, we reconstruct the pathways connecting query construction, visual evidence retrieval, and answer readout. This enables us to investigate how VRHs influence predictions and the hallucination mechanism behind their behavior. Appendix C.3 shows how similar object-search queries interact with available visual evidence: when the target is absent, retrieval can shift toward a semantic confuser, whose evidence can support a confident but incorrect presence decision. This understanding motivates TRACE's retrieval correction and the initial targeted training in Appendix C.3.1. Our motivation therefore extends beyond proposing another mitigation method: we seek to advance the understanding of VLM hallucinations and establish a mechanistic foundation for interventions targeting the computations that produce them.

[1] Jing Bi et al. Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach. CVPR, 2025. arXiv:2412.18108.

[2] Chanho Park, Daehyeon Choi, Jihyun Lee, and Minhyuk Sung. Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information. arXiv:2608.27417, 2026.

Weakness 2: The contribution beyond external detection is not isolated. At inference time, TRACE receives object-presence confidence and bounding boxes from an external detector. For object-existence questions, this information already closely matches what the model must predict. Without detector-only and detector-informed prompting baselines using the same external information, it is difficult to attribute the gains specifically to causal head intervention.

Response: Thanks for raising this concern. To isolate the contribution of causal head intervention, we have added a comparison against both a YOLO26-only baseline and YOLO26-informed prompting using the same detector signals to the revised manuscript (Table 5). The results are reproduced below.

MethodRandomPopularAdversarialOverall
Greedy88.4 / 88.584.8 / 85.578.1 / 80.383.8 / 84.8
YOLO26-only (raw)85.7 / 83.485.2 / 82.984.9 / 82.685.3 / 83.0
YOLO26-informed prompting90.2 / 90.782.9 / 85.176.0 / 80.283.0 / 85.3
TRACE90.7 / 89.989.7 / 88.988.2 / 87.689.5 / 88.8

These comparisons indicate that detector cues alone do not account for TRACE’s overall gains, while also showing that the advantage varies by split and metric.

AuthorsOpenReview ↗

Response to Reviewer MJZe — Submission 55074 (Part 2)

Question 1: How does TRACE ensuress preserving general multimodal understanding? Object recall decreases on both models in the captioning evaluation, making it important to establish whether reduced hallucination comes at the cost of useful visual content. Results on general multimodal benchmarks would directly affect this assessment.

Response: We have added results on 4 MME's subtasks for LLaVA-1.5-7B (Appendix B.3, Table 8). We retain the original TRACE configuration and apply the intervention only at the object position.

MethodExistenceCountPositionColor
Greedy185133.33101.67135
TRACE195128.33101.67140

These results show that TRACE largely preserves performance on the evaluated MME subtasks, with only modest changes: Existence and Color improve, Count decreases slightly, and Position is unchanged.

To better understand this behavior, we extended our causal analysis to MME's Color and Count tasks (Appendix B.3). The results suggest a shared visual-grounding pathway alongside task-specific retrieval of color or quantity evidence. Grounding is therefore only part of the computation; the final answer also requires integrating these sources of information. TRACE reinforces object-level visual grounding, so improving this component alone need not improve the final decision. We leave a detailed mechanistic analysis of how the model combines information from these components to future work, as it lies beyond the scope of this study.

Question 2: Does the identified three-stage circuit persist in newer VLMs and more challenging hallucination settings? The current evidence from LLaVA-1.5-7B and Qwen2-VL-7B on POPE, BEAF, and CHAIR leaves its relevance to current architectures, training recipes, and tasks uncertain. Also, the benchmarks used in this article are relatively outdated, more evaluatoins on frontier benchmarks would address this concern to a great extent

Response: Thank you for your question. We have added experiments on Qwen3-VL-8B-Instruct in Appendix B.2.2, including causal circuit analysis and TRACE evaluation on POPE-COCO. The analysis provides evidence for the same three-stage organization: query construction, visual evidence retrieval, and answer readout. The results are reported in Table 7 and reproduced below.

POPE-COCO splitGreedy AccuracyGreedy F1TRACE AccuracyTRACE F1
Random92.191.590.589.6
Popular88.788.489.488.5
Adversarial86.686.588.787.8
Overall89.188.889.588.6

We also conducted additional experiments on POPE-AOKVQA, POPE-GQA, AMBER-Existence, and Causal-HalBench, using Grounding DINO and SAM3 alongside YOLO26. The results from Table 6 are below, with full results and experimental settings provided in Appendix F.5. Scores are percentages; POPE averages the adversarial results across COCO, AOKVQA, and GQA.

MethodPOPE Adversarial (avg.) AccPOPE Adversarial (avg.) F1AMBER-Existence AccCausal-HalBench AccCausal-HalBench F1
Greedy70.676.470.978.678.1
TRACE + YOLO2679.278.485.884.779.9
TRACE + Grounding DINO81.182.290.785.883.3
TRACE + SAM382.983.990.687.684.5

Together, these results support the persistence of the three-stage circuit in Qwen3-VL and the effectiveness of TRACE on additional hallucination benchmarks.

AuthorsOpenReview ↗

Response to Reviewer MJZe — Submission 55074 (Part 3)

Question 3: It seems that Object-existence QA and captioning use separately discovered heads and different intervention procedures: direct answer intervention versus candidate verification and logit adjustment. How would TRACE handle general or mixed multimodal queries involving descriptions, attributes, counting, and relations? Is there a unified inference procedure, or is its applicability currently restricted to the two evaluated task formats?

Response: Thanks for asking us to clarify the scope of TRACE. Our two variants use task-specific head discovery and inference procedures, while following the same mechanism of correcting object-level visual retrieval. Their differences reflect how this shared correction is applied to each task format.

Causal circuit studies show that related tasks can share computational components while differing in their complete pathways and how those components are combined [3, 4]. Shared computation therefore does not guarantee that the full circuit or an intervention transfers unchanged across tasks.

Our focus is object-existence grounding and its role in hallucination, evaluated through existence QA and object verification during captioning. As discussed in our response to Question 1 and Appendix B.3, other tasks can share visual grounding, but forming their answers also requires combining it with task-specific information about attributes, quantities, or relations. TRACE currently provides two inference procedures built around the shared object-level retrieval mechanism. A unified procedure for arbitrary mixed multimodal queries would require further analysis of these additional computations and lies beyond the scope of this study. We leave this extension to future work.

[3] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit Component Reuse Across Tasks in Transformer Language Models. ICLR, 2024. arXiv:2310.08744.

[4] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. NeurIPS, 2023. arXiv:2305.00586.


We sincerely thank you for your thoughtful review. We hope the revised manuscript and the clarifications provided above have addressed your concerns effectively. If our responses have satisfactorily resolved your concerns, we would greatly appreciate it if you could consider updating your review score. However, if you have any remaining concerns or require further clarification, please do not hesitate to let us know.

Reviewer_hMceOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TRACE, a training-free method for mitigating visual hallucinations based on causal analysis. The authors first investigate the internal mechanisms underlying object-existence predictions in MLLMs and identify a related circuit, in which Visual Retrieval Heads (VRHs) play a key role in hallucination: when these heads attend to incorrect image regions or fail to allocate sufficient attention to the correct regions, the model may produce erroneous predictions, including overconfident hallucinations. Based on this observation, the authors introduce TRACE, which uses an external object detector to guide the attention patterns of VRHs and thereby mitigate hallucinations. For object-existence QA, TRACE directly modifies the queries of VRHs at the object-name token. For image captioning, TRACE adjusts the logits of candidate object tokens based on the effects of the same intervention.

Experiments on LLaVA-1.5-7B and Qwen2-VL-7B show that TRACE effectively reduces hallucinations and outperforms existing baselines.

Critical strengths
  • The paper provides an in-depth causal analysis of the internal mechanisms underlying object hallucinations and attempts to intervene at the source of these errors, rather than merely adjusting the final outputs.

  • The proposed method is well-motivated and principled, with its design grounded in the findings of the causal analysis.

Critical weaknesses
  • The paper lacks an evaluation of general multimodal capabilities. Since TRACE modifies the model's internal attention patterns, it is important to verify that these interventions do not negatively affect the model's original general capabilities.
Questions for authors
  • When the external detector misses an object that is actually present, TRACE appears to steer the model toward predicting its absence. How robust is TRACE to such detection failures? Could this also contribute to the reduced object recall observed in the CHAIR results?

  • How well would TRACE generalize to scenarios where an object name is generated without requiring visual grounding? For example, if the model mentions an object while answering a question unrelated to the image, could TRACE introduce unnecessary logit adjustments? My intuition is that the retrieval-based verification mechanism might naturally limit such interference, but I am curious whether the authors have explored this setting. This question is mainly motivated by an interest in the method's generalizability rather than a concern about its performance on the evaluated tasks.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hMce — Submission 55074

Thank you for reviewing our paper and for offering constructive feedback. We truly value your effort and have addressed your concerns and questions below.

Weakness 1: The paper lacks an evaluation of general multimodal capabilities. Since TRACE modifies the model's internal attention patterns, it is important to verify that these interventions do not negatively affect the model's original general capabilities.

Response: Thank you for highlighting this. We now report LLaVA-1.5-7B results on MME's Existence, Count, Position, and Color subtasks in Appendix B.3 (Table 8). All TRACE settings remain unchanged, with correction applied only at the queried-object position.

MethodExistenceCountPositionColor
Greedy185133.33101.67135
TRACE195128.33101.67140

Performance remains broadly comparable across these evaluated subtasks: TRACE improves Existence and Color, slightly lowers Count, and leaves Position unchanged.

We also conducted causal analyses of MME's Color and Count tasks (Appendix B.3). These analyses suggest that shared object-grounding computations operate alongside task-specific retrieval of color or quantity evidence. Forming the final answer also requires integrating these sources of information. TRACE focuses on the grounding component, which may explain its modest and mixed effects on these tasks. How the model combines these computations is beyond the scope of this paper.

Question 1: When the external detector misses an object that is actually present, TRACE appears to steer the model toward predicting its absence. How robust is TRACE to such detection failures? Could this also contribute to the reduced object recall observed in the CHAIR results?

Response: Thanks for raising this concern. Across 3,000 POPE-COCO adversarial queries, TRACE corrects 478 initially incorrect answers while making 175 (162 coming from detector errors) previously correct answers incorrect. This gives a net gain of 303 correct answers, increasing accuracy from 78.1% to 88.2%. Overall, TRACE corrects more errors than it introduces.

However, the impact of detector errors on accuracy is undeniable. We acknowledge this limitation and will address it in our future work. Beyond detector-guided correction, our causal analysis provides a foundation for directly repairing the model's retrieval mechanism. Appendix C.3 suggests that similar object-search queries can retrieve evidence from semantic confusers when the target is absent. As an initial proof of concept, finetuning Query Construction Heads' O and V matrices of improves POPE-COCO performance (Appendix C.3.1, Table 13).

Question 2: How well would TRACE generalize to scenarios where an object name is generated without requiring visual grounding? For example, if the model mentions an object while answering a question unrelated to the image, could TRACE introduce unnecessary logit adjustments? My intuition is that the retrieval-based verification mechanism might naturally limit such interference, but I am curious whether the authors have explored this setting.

Response: Thank you for this interesting question. We had not evaluated this setting previously. Mechanistically, a circuit's contribution depends on the computation required by the task. Hanna et al. [1] show through causal analysis that some related tasks reuse their greater-than circuit, while other mathematical tasks rely on different heads and MLPs. Following this task-dependent view, we hypothesize that image-independent factual answers may be less sensitive to TRACE when visual retrieval contributes little to their computation.

We tested two factual questions with two images, one containing a cat and one without. We used exactly the LLaVA-1.5-7B configuration reported in the paper, applying TRACE at the token "cat". LD denotes logit(Yes) - logit(No).

ImageQuestionOriginal LDTRACE
Cat presentIs a cat a mammal?+8.000+8.000
Cat presentIs a cat a reptile?−4.500−4.750
Cat absentIs a cat a mammal?+4.250+4.500
Cat absentIs a cat a reptile?−3.625−3.500

All four answers remained correct and unchanged, providing preliminary evidence that TRACE can modify visual retrieval without changing answers to image-independent questions.

[1] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. NeurIPS, 2023. arXiv:2305.00586.


We hope the revised manuscript and the clarifications provided above have addressed your concerns. If our responses have resolved your concerns, we would greatly appreciate it if you could consider updating your review score. If you have any remaining concerns or require further clarification, please do not hesitate to let us know.

Thank you for your detailed responses and the additional experiments. The new results have alleviated some of my concerns. Overall, I believe my original rating remains the most appropriate assessment of this work.

AuthorsOpenReview ↗

Thanks for your feedback

We are very pleased to hear that our responses and additional experiments have addressed some of your concerns. We sincerely appreciate your valuable and constructive feedback, which has helped us significantly strengthen the paper. Thank you again for your thoughtful review and for taking the time to engage with our work.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response and Summary of Changes

We would like to thank all reviewers for the fruitful discussion and their detailed reviews. We would like to summarize the following main points from the rebuttal process.

  1. Strengths of the paper

    • Depth of causal analysis. All reviewers appreciated the depth of our causal analysis in object hallucinations and described our analysis as an “in-depth causal analysis” [hMce], “sound and detailed” [MJZe], “thorough and convincing” [EUUK].

    • Principled method design. TRACE was recognized as “well-motivated and principled,” with its design “grounded in the findings of the causal analysis” [hMce].

    • Strong experimental results. Reviewers noted “Consistent improvements within the evaluated setting” [MJZe] and “substantial hallucination reduction with only a moderate reduction in object recall” [EUUK].

    • Practical inference-cost evaluation. The resource comparison was described as “valuable for an inference-time method,” addressing concerns about the captioning variant’s overhead [EUUK].

  2. Concerns and additional results

    We have revised the manuscript in response to their feedback, with new and changed text marked in blue. The main revisions are summarized below.

    • Detector-output controls (Reviewer MJZe).

      Concern: Whether TRACE’s gains come from causal head intervention or simply from access to external detector information.

      Added: YOLO26-only prediction and YOLO26-informed prompting baselines using the same detector signals (Section 5.3, Table 5; Appendix D.3.1).

      Result: TRACE outperforms both controls in overall Accuracy and F1 on POPE-COCO, showing that detector cues alone do not reproduce its overall gains.

    • Alternative grounding models and additional benchmarks (Reviewers EUUK and MJZe).

      Concern: Dependence on YOLO26’s closed vocabulary and limited evidence beyond COCO and the original benchmarks.

      Added: Experiments with pretrained Grounding DINO and SAM3 on POPE-AOKVQA, POPE-GQA, AMBER-Existence, and Causal-HalBench, alongside COCO (Section 5.3, Table 6; Appendix F.5).

      Result: Both alternatives improve the aggregate POPE results and performance on AMBER-Existence and Causal-HalBench over TRACE with YOLO26. This supports compatibility with the evaluated open-vocabulary grounding models and shows that better-suited grounding signals can further improve TRACE.

    • Experiments on a newer VLM (Reviewers EUUK and MJZe).

      Concern: Whether the identified mechanism and TRACE extend to more recent VLMs.

      Added: Causal circuit analysis and TRACE evaluation on Qwen3-VL-8B-Instruct (Appendix B.2.2, Figures 12–13 and Table 7).

      Result: The analysis provides evidence for the same three-stage organization with model-specific heads. TRACE improves both metrics on Popular and Adversarial; overall Accuracy increases slightly, while Random performance and overall F1 decrease. Together, these findings support the generalization of the identified mechanism and TRACE to a more recent VLM architecture.

    • General multimodal evaluation and analysis (Reviewers hMce and MJZe).

      Concern: Whether TRACE preserves general multimodal capabilities and how its retrieval correction relates to tasks beyond object existence.

      Added: Evaluation on MME’s Existence, Count, Position, and Color subtasks using the original LLaVA TRACE configuration, together with Color/Count path patching and attention analysis (Appendix B.3, Figures 14–16 and Table 8).

      Result: Performance remains broadly comparable on these evaluated subtasks. The analyses suggest shared grounding alongside task-specific computations, helping explain the limited and mixed effects of object-level retrieval correction on these tasks.

    We also summarize two further clarifications from our individual responses below.

AuthorsOpenReview ↗

Additional Clarifications (1/2)

  • Novelty of visual retrieval head discovery (Reviewer MJZe).

    Concern: Whether identifying specialized Visual Retrieval Heads is itself novel, given prior studies.

    Response: Prior work [1, 2] has established the existence and importance of Visual Retrieval Heads (VRHs). The cited works primarily focus on identifying and characterizing visual heads through attention patterns and causal validation; they do not provide a comprehensive, interpretable view of how VLMs process visual queries, retrieve visual evidence, and come up with the final prediction.

    In contrast, while identifying visual heads is an important part of our paper, our central goal is to reconstruct the computation flow through which a VLM processes visual evidence and produces an object-existence decision. To the best of our knowledge, our work provides the first end-to-end, human-interpretable account of this flow in the context of object hallucination (Fig. 2).

    Therefore, the novelty of our causal analysis does not lie in rediscovering Visual Retrieval Heads by themselves, but in revealing their functional roles within this computation flow. Beyond Visual Retrieval Heads, we also identify Query Construction Heads, which form object-specific retrieval signals, and Answer Mover Heads, which carry the resulting evidence toward the answer.

    Most importantly, our work connects the computation flow of VLMs to the mechanism of hallucination. Specifically, we demonstrate how and where VLMs retrieve false evidence from semantically similar but incorrect image regions, introducing hallucination into the generation process (Fig. 5; Appendix C.3). This further motivates a principled, targeted decoding process to mitigate hallucination. Identifying VRHs alone would not reveal this hallucination mechanism or specify how to intervene; both require tracing how evidence flows through the model.

    Park et al. [2] first appeared on arXiv on August 27, 2026. The ICLR 2027 Reviewer Guidelines do not require comparisons with contemporaneous or arXiv-only work and state that missing such comparisons cannot be a basis for rejection. We will nevertheless cite and discuss this work in the updated manuscript.

[1] Jing Bi et al. Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach. CVPR, 2025. arXiv:2412.18108.

[2] Chanho Park, Daehyeon Choi, Jihyun Lee, and Minhyuk Sung. Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information. 2026. arXiv:2608.27417.

AuthorsOpenReview ↗

Additional Clarifications (2/2)

  • Robustness to detector errors (Reviewers hMce and EUUK).

    Concern: Whether detector failures misdirect TRACE.

    Response: We acknowledge that detector failures can misdirect TRACE. On 3,000 POPE-COCO adversarial queries, however, TRACE corrects 478 baseline errors while introducing 175 new errors, yielding a net gain of 303 correct answers. As discussed above, our broader goal is to address the underlying hallucination mechanism. Developing a targeted method that does not rely on external detectors is therefore a key step toward this goal. As an initial proof of concept, V/O hybrid tuning of four Query Construction Heads improves accuracy from 78.10% to 85.30% (Appendix C.3.1, Table 14). We will pursue this direction toward mechanism-targeted, detector-free inference in future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.