- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VisDelta, a training-free inference-time framework that keeps the underlying LVLM frozen to mitigate object hallucinations. VisDelta introduces two core components: (1) Dual-branch Contrastive Trajectory (DCT), which compares normal decoding with a visually blocked reference branch to locate the layer where visual influence diminishes; and (2) Coarse-to-Fine Evidence Reinforcement (CFER), which re-encodes query-relevant image crops and dynamically injects visual evidence into FFN computations via Contribution-Controlled Reinforcement (CCR). Evaluated across different LVLMs on POPE, CHAIR, MME, and MMBench, VisDelta demonstrates consistent hallucination reduction—notably boosting POPE F1 and CHAIR on LLaVA-1.5—while preserving general multimodal capabilities.
Critical strengths
- The paper insightfully analyzes the limitations of relying on conventional attention/entropy signals, proposing a mechanistically self-consistent layer-wise probing mechanism. The underlying rationale is thoroughly derived, and the figures are high-quality and informative.
- The proposed method demonstrates consistent improvements across key hallucination benchmarks (CHAIR and POPE). Notably, the inclusion of general multimodal benchmarks (MME and MMBench) serves as a thoughtful sanity check, confirming that hallucination mitigation does not compromise general multimodal reasoning capabilities.
Critical weaknesses
- The backbone models used in the experimental section are all from 2023 and 2024, while the new model has already iterated many versions (such as QWEN 3.8). We have difficulty knowing whether this method is still effective on the new model.
- Although the method proposed in the article is effective, it also brings significant computational overhead. However, the paper did not report relevant indicators such as running time and computing power consumption. Therefore, it is difficult to determine whether the gain and efficiency are balanced.
- The method proposed in the article lacks novelty. The author combines the idea of contrastive decoding, and the methods of cropping images and assigning weights have been mentioned in numerous articles.
- The author seems to consider a decrease in visual influence as a reliable indicator that requires visual evidence reinforcement. However, it is difficult to prove in the experimental section that it is truly related to hallucinations. Assuming the following scenario, the question is' Is the image a bicycle or a motorcycle? 'The image looks very similar to a motorcycle, but the model generates' bicycle' due to language priors. At this point, even if the model generates hallucination, visual information may still have a strong impact on the internal representation of the model. That is to say, DCT seems to only reflect whether the model is affected by visual information, but it is difficult to reflect whether the visual information received by the model is correct
Questions for authors
- Please provide concrete efficiency benchmarks in the Rebuttal, including inference latency, throughput, and additional computational/memory overhead compared with the baseline and competing methods.
- Please evaluate the method on newer and stronger LVLM backbones to demonstrate that the proposed approach generalizes beyond the relatively older models currently considered.