- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TRACE, a training-free method for mitigating hallucinations in LVLMs. TRACE propagates contribution-based provenance estimates for visual inputs, user queries, and generation history across layers. During decoding, it detects abnormal increases in history-relative-to-visual influence and selectively reinforces the estimated visual-origin contribution. The experimental results demonstrate the effectiveness of it in mitigating hallucinations.
Critical strengths
- Well-motivated perspective. Distinguishing information origin from its current token position is a useful perspective, since visual information can propagate through textual representations. This motivates looking beyond direct contributions from image-token positions.
- Coherent method design. TRACE connects provenance estimation, online detection, and adaptive intervention within a single framework, without additional training or an extra model forward pass.
Critical weaknesses
- Insufficient causal validation. The propagated weights provide attribution under the proposed rule, but do not establish that textual representations retain task-relevant visual semantics or that provenance takeover causes hallucinations.
- Unclear specificity of visual reinforcement. The paper does not establish that the gains are specific to reinforcing text positions assigned high visual-provenance weights, nor does it show that reinforcing randomly selected positions is ineffective.
- Reporting issues. Figure 1 lacks detailed model and experimental settings (e.g., sample and object counts), and the relative latency calculation for CRG in Table 4 is incorrect.
- Incomplete discussion of related work. Prior work on explicit [1, 2] and implicit [3] cross-modal information flow is not discussed. The paper should clarify how its provenance formulation advances beyond these studies.
Questions for authors
-
Could the authors add activation patching experiments, using controlled corruption and restoration, to test whether text positions assigned high visual provenance causally contribute to visually grounded predictions?
-
Could the authors compare against reinforcing randomly selected text positions, matching the number of contributing positions, intervention timing, and perturbation magnitude, to test whether this random control also reduces hallucinations?
References
[1] Cross-modal information flow in multimodal large language models. CVPR 2025.
[2] Same task, different circuits: Disentangling modality-specific mechanisms in vlms. NeurIPS 2025.
[3] Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow. ICML 2026.
3 comments
Follow-up on Rebuttal Responses
Dear Reviewer,
We have provided detailed responses to your concerns and would appreciate it if you could take a look when you have a chance.
As the discussion period is coming to an end, we would be grateful for any feedback on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.
Best regards,
The Authors
Thank you for the detailed rebuttal and additional experiments, which alleviate some of my concerns; I now consider the paper borderline, with no preference toward acceptance or rejection.
Reply to reviewer pT86
We thank the reviewer for the constructive feedback. We address the concerns and questions below.
W1 & Q1. Causal validation of provenance attribution
We appreciate this concern. TRACE estimates source-level contributions rather than claiming an exact causal decomposition of hidden representations. Our propagation mechanism tracks estimated information origins across Transformer layers, allowing visual-origin contributions to remain distinguishable even after being integrated into textual representations.
To strengthen the empirical validation, we have revised Figure 1 to present separate provenance trajectories for LLaVA-1.5-7B, Qwen-VL-Chat, and Qwen2.5-VL on 500 CHAIR images. Object mentions are extracted from each model's generated descriptions and classified as grounded or hallucinated using ground-truth annotations. We also provide real qualitative examples in Supplementary Section A.4, illustrating hallucinated object mentions and their correction after TRACE intervention.
More importantly, we have added an activation-patching experiment in Supplementary Section A.5 to directly examine the causal relevance of estimated visual provenance. Using 200 clean–corrupted image pairs from MS COCO, we restore clean activations at text positions selected by high visual provenance, low visual provenance, or random selection, while keeping the number of patched positions identical. High-visual-provenance positions consistently achieve substantially greater prediction recovery than both controls, particularly in middle-to-late Transformer layers. These results provide intervention-based evidence that TRACE identifies textual representations that causally contribute to visually grounded predictions, although they do not establish an exact decomposition of visual information.
W2 & Q2. Specificity of visual reinforcement
We appreciate this suggestion. We have added two ablations: random-token reinforcement and random gating, to examine whether TRACE's improvements arise from provenance-guided reinforcement and takeover-based intervention rather than arbitrary reinforcement.
Random-token reinforcement yields only a modest reduction in CHAIRs while decreasing Recall from 77.1 to 73.5, suggesting that arbitrary reinforcement can suppress relevant visual content. Similarly, random gating achieves substantially weaker hallucination mitigation than takeover-guided intervention. These results support the importance of selecting representations according to their estimated visual provenance and using the takeover signal to guide intervention.
W3. Reporting issues
We thank the reviewer for identifying these issues. In the revised version, we have updated Figure 1 to specify the evaluated models, dataset, sample size, and object-mention analysis protocol. We have also corrected the relative-latency calculation for CRG in Table 4.
W4. Distinction from related cross-modal information-flow studies
We thank the reviewer for highlighting these works. While [1–3] also investigate cross-modal information flow, their attribution mechanisms and objectives differ from TRACE.
Specifically, [1] uses attention knockout to identify important information-transfer pathways between visual and textual token positions, measuring their importance through changes in model predictions. However, identifying information transfer between positions does not directly quantify its original source after cross-token mixing. TRACE instead propagates source-level attribution across layers, allowing visual-origin information to remain attributed to its original source even when carried by textual representations.
[2] compares computational circuits for equivalent visual and textual tasks, using back-patching to address differences in modality-specific processing. It does not estimate how source contributions evolve during autoregressive generation. [3] investigates emotion-related information flow using a steering-vector-based causal attribution framework, whereas TRACE estimates source provenance directly during decoding without constructing separate steering vectors.
More importantly, TRACE uses these source-provenance estimates to investigate provenance takeover and guide adaptive hallucination mitigation. Unlike [1–3], our method connects the evolving contributions of visual inputs, user queries, and generation history to an online detection signal and corresponding intervention strategy.