- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes hallucination reduction method in multimodal LLMs including LVLMs and LALMs. The paper argues that the hallucination occurs when the model do not attend enough to the perceptual inputs. This motivates training-free LIME method, in which they optimize key-value representations so that the models rely more on the perceptual inputs rather than textual inputs.
Critical strengths
The method is generalizable across various LLMs including LVLMs and LALMs.
Critical weaknesses
Major Concerns
-
The claims in lines 48-79 are weakly supported. I cannot find any quantitative results in the main body that link perceptual relevance to hallucination. Moreover, even if supporting results are available, the findings themselves do not appear to offer substantial novelty compared to prior papers. Also, line 204 states that this relation is analyzed in Section 4.3, but Section 4.3 contains no such analysis. I can only find the table about the LIME method itself.
-
Limited practicability as training-free method: The overhead is too heavy for the gains. The main results use 7 steps, which causes a 9.43x slowdown (Table A.9). Compared to prior methods, the improvement is small to justify this cost. In addition, no video or other long-input benchmark is evaluated. I think the cost in both time and memory should grow quickly with the number of perceptual tokens.
-
The explanation for why localization emerges without supervision is not convincing. The objective alone should favor a uniform spread over perceptual tokens. Potential explanation is needed.
Minor Concerns
- Sections 4.1 and 4.2 claim evaluation on five vision and three audio models, but results for several of them appear only in the appendix. This makes the main results hard to follow.
- Numeric precision is inconsistent across tables (xx, xx.x, xx.xx). Please use a uniform format.
- The paper states that some baseline methods are "not applicable across all architectures" (lines 302-303), but contrastive decoding methods such as VCD should be applicable to Qwen2.5-VL and Qwen2.5-Omni, and others. Such inaccurate statements raise doubts about whether the comparison is fair.
Questions for authors
- The baseline comparisons are mainly on older models (LLaVA-1.5-7B, Qwen-VL-Chat, SALMONN-7B). Does LIME generalize to more recent, stronger models such as Qwen3-VL?
- Is there an ablation without the KL term?
- How do the slowdown and peak memory scale with sequence length (e.g., long video inputs)?
- Is there any quantitative result supporting the claim in line 53 that both patterns are "most pronounced on samples the model gets wrong"?
5 comments
Response to Minor Concern and Questions by Authors
Minor Concerns:
- We agree that the presentation of our experimental results could be clearer. Due to space limitations, we included representative results in the main paper and placed additional evaluations in Appendix B.3. To improve clarity, we have revised the references to these additional results throughout Sections 4.1 and 4.2.
- We thank the reviewer for pointing out this inconsistency. We have fixed the numerical formatting across all tables in the revised manuscript
- We thank the reviewer for pointing out this distinction. Our goal was to provide a comprehensive comparison against available training-free methods, including results where compatible and reproducible implementations were available. The limitations regarding baseline applicability primarily concern implementation availability rather than fundamental architectural incompatibility. For example, VCD is conceptually applicable to Qwen2.5-VL and Qwen2.5-Omni, but its publicly released implementation does not directly support these architectures and would require further adaptation. We have clarified this distinction in the revised manuscript and believe that the existing comparisons remain representative of established training-free approaches, providing a meaningful basis for evaluating LIME across different models and benchmarks.
Questions:
- To further evaluate the generality of LIME, we have added experiments with Qwen3-VL on the POPE benchmark using both MSCOCO and A-OKVQA (Table A.5). On MSCOCO, LIME improves accuracy from 91.47% to 93.31% and F1 from 91.28% to 93.25%. On A-OKVQA, accuracy improves from 93.88% to 94.78% and F1 from 93.71% to 94.59%. Our original evaluation also included MiniCPM-o 4.5, released in 2026, for which LIME consistently improves accuracy and F1 on both datasets. Together, we believe that these results provide further evidence of LIME's generalization to newer multimodal architectures
- We have added an ablation (lines 1026-1033) removing the KL term entirely (), while keeping all other optimization settings unchanged. As reported in the new Table A.7, LIME without KL achieves 88.70% accuracy on LLaVA and 62.10% on Qwen2-Audio, compared with baseline accuracies of 83.49% and 56.19%, respectively. Including KL regularization further improves accuracy to 90.27% and 63.36%. These results show that the relevance objective alone provides substantial improvements, while KL regularization yields additional gains by limiting deviations from the original output distribution.
- As discussed in our response to Major Concern 2, LIME introduces computational overhead that may increase with input sequence length. While our efficiency analysis in Section 4.5 and Appendix D examines the effects of optimization budget and intervention scope, we have not directly evaluated scalability to long-form inputs such as video. We acknowledge this a in the revised manuscript.
- As discussed in our response to Major Concern 1, Appendix C.1 provides quantitative evidence linking relevance distribution to hallucination behavior. Table A.9 presents a per-sample analysis showing that corrected predictions exhibit substantially larger increases in perceptual utilization and relevance localization than predictions that remain hallucinated. Complementing this analysis, Table A.10 presents controlled interventions examining how the direction and target of relevance optimization affect prediction accuracy.
Official Comment by Authors
Thank you again for your review. We believe that our rebuttal addresses each of the questions and concerns raised in your review. If any points remain unclear or if further clarification would be helpful, we would be happy to provide additional details during the remaining discussion period.
If you feel that your concerns have been adequately addressed, we would be grateful if you would consider updating your score accordingly.
Response to Major Concerns by Authors
Thank you for your comments. We appreciate your response and would like to address the points you raised. We also uploaded a revised version of the paper.
Major Concerns:
We would like to clarify the evidence linking relevance distribution to hallucination behavior. Section 4.3 (Table 4) examines perceptual utilization and relevance localization, while Appendix C.1 provides complementary behavioral analyses, including per-sample comparisons and controlled interventions on both positive and negative samples (Tables A.9 and A.10).
First, Table A.9 shows that corrected predictions exhibit larger increases in perceptual utilization (+0.121 vs. +0.031) and relevance localization (+0.147 vs. +0.026) than predictions that remain hallucinated. Second, Table A.10 presents controlled interventions showing that increasing perceptual relevance improves accuracy (83.24% to 90.44%), while suppressing it reduces accuracy (82.65%), particularly when targeting query-relevant regions (79.82%). Random interventions have negligible effect (83.35%). Together, these results provide observational and interventional evidence linking relevance distribution to hallucination behavior. We have clarified this distinction from the modality utilization analysis in Section 4.3 in the revised manuscript.
The novelty of our approach lies in using prediction-specific relevance to both characterize and mitigate multimodal hallucinations. While existing methods primarily rely on indirect signals, such as attention weights or output changes under perturbed inputs [1, 2, 3], LIME uses LRP to quantify the contributions of perceptual and textual tokens to model predictions. To the best of our knowledge, we are the first to extend LRP beyond post-hoc explainability by using relevance directly as an inference-time optimization signal to mitigate multimodal hallucinations.
We agree that LIME introduces additional computational overhead due to its iterative relevance-guided optimization. To better characterize the accuracy-latency trade-off, we have expanded the efficiency evaluation to include three-step configurations, reporting measured throughput, slowdown, peak memory, and accuracy (Table 5 in the revised manuscript). On LLaVA, three steps achieve 88.65% accuracy at a 3.33× slowdown, compared with 90.27% at 9.43× using seven steps. Similarly, on Qwen2-Audio, three steps achieve 61.00% accuracy at 3.21×, compared with 63.36% at 9.20× using seven steps. We use higher learning rates for the lower-budget configurations, as we believe this enables more effective optimization with fewer steps, while acknowledging the need for hyperparameter selection. While methods such as MemVR incur lower computational costs, LIME allows users to balance accuracy and latency through its optimization budget and demonstrates effectiveness across both vision and audio models.
Regarding long-input scalability, LIME has not yet been evaluated on long-form inputs such as video, where the computational overhead may increase with sequence length. Improving the efficiency of relevance-guided optimization therefore remains an important direction for future work. We have moved the efficiency analysis and limitations discussion into the main paper, to provide a clearer account of the measured computational costs, configurable trade-offs, and the remaining scalability challenges.
[1] "OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation", Huang el al., CVPR 2024
[2] "Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding", Leng et al., CVPR 2024.
[3] "Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding", Hsu et al, 2025