acceptodds
Withdrawn from ICLR 2027

Contrastive Trajectory-Guided Visual Evidence Reinforcement for Hallucination Mitigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Visual hallucination remains a major obstacle to the reliable deployment of large vision-language models (LVLMs), as fluent responses may contain objects, attributes, or relations that are not adequately grounded in the input image. Existing inference-time methods commonly rely on predictive uncertainty or direct attention to visual tokens to determine when and where to intervene. However, these signals do not isolate the effect of visual access on the current token after image information has propagated through the decoder. We propose VisDelta, a training-free method that uses controlled visual-access contrast to identify where visual influence begins to diminish and selectively reinforces query-relevant image evidence. Specifically, Dual-branch Contrastive Trajectory (DCT) compares a standard decoding branch with a matched reference branch in which non-visual queries are prevented from attending to visual keys, and selects an intervention layer based on a sustained decline in their normalized current-token attention-output difference. At the selected layer, Coarse-to-Fine Evidence Reinforcement (CFER) localizes query-relevant image regions and re-encodes their original pixels with the frozen vision encoder. Contribution-Controlled Reinforcement (CCR) then injects the retrieved local features into the feed-forward computation and adaptively scales the update based on their source-region contributions. Experiments on CHAIR and POPE across multiple LVLM backbones demonstrate that VisDelta effectively reduces object hallucination and outperforms existing inference-time baselines. Results on MME and MMBench further indicate that these improvements are achieved while preserving general multimodal capabilities, and controlled ablations validate the respective contributions of layer selection, regional evidence retrieval, and contribution-controlled integration.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal large language models, Object hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VcZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VisDelta, a training-free inference-time framework that keeps the underlying LVLM frozen to mitigate object hallucinations. VisDelta introduces two core components: (1) Dual-branch Contrastive Trajectory (DCT), which compares normal decoding with a visually blocked reference branch to locate the layer where visual influence diminishes; and (2) Coarse-to-Fine Evidence Reinforcement (CFER), which re-encodes query-relevant image crops and dynamically injects visual evidence into FFN computations via Contribution-Controlled Reinforcement (CCR). Evaluated across different LVLMs on POPE, CHAIR, MME, and MMBench, VisDelta demonstrates consistent hallucination reduction—notably boosting POPE F1 and CHAIR on LLaVA-1.5—while preserving general multimodal capabilities.

Critical strengths
  1. The paper insightfully analyzes the limitations of relying on conventional attention/entropy signals, proposing a mechanistically self-consistent layer-wise probing mechanism. The underlying rationale is thoroughly derived, and the figures are high-quality and informative.
  2. The proposed method demonstrates consistent improvements across key hallucination benchmarks (CHAIR and POPE). Notably, the inclusion of general multimodal benchmarks (MME and MMBench) serves as a thoughtful sanity check, confirming that hallucination mitigation does not compromise general multimodal reasoning capabilities.
Critical weaknesses
  1. The backbone models used in the experimental section are all from 2023 and 2024, while the new model has already iterated many versions (such as QWEN 3.8). We have difficulty knowing whether this method is still effective on the new model.
  2. Although the method proposed in the article is effective, it also brings significant computational overhead. However, the paper did not report relevant indicators such as running time and computing power consumption. Therefore, it is difficult to determine whether the gain and efficiency are balanced.
  3. The method proposed in the article lacks novelty. The author combines the idea of contrastive decoding, and the methods of cropping images and assigning weights have been mentioned in numerous articles.
  4. The author seems to consider a decrease in visual influence as a reliable indicator that requires visual evidence reinforcement. However, it is difficult to prove in the experimental section that it is truly related to hallucinations. Assuming the following scenario, the question is' Is the image a bicycle or a motorcycle? 'The image looks very similar to a motorcycle, but the model generates' bicycle' due to language priors. At this point, even if the model generates hallucination, visual information may still have a strong impact on the internal representation of the model. That is to say, DCT seems to only reflect whether the model is affected by visual information, but it is difficult to reflect whether the visual information received by the model is correct
Questions for authors
  1. Please provide concrete efficiency benchmarks in the Rebuttal, including inference latency, throughput, and additional computational/memory overhead compared with the baseline and competing methods.
  2. Please evaluate the method on newer and stronger LVLM backbones to demonstrate that the proposed approach generalizes beyond the relatively older models currently considered.
Reviewer_Y7ebOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VisDelta, a training-free method for mitigating hallucinations in LVLMs by selectively reinforcing visual evidence during decoding. Specifically, the authors first introduce Dual-branch Contrastive Trajectory, which compares decoding trajectories with and without direct access to visual tokens to identify suitable intervention layers. At the selected layer, the method retrieves relevant visual patches based on their similarity to the current query, groups them into image regions, and re-encodes the corresponding crops to obtain more fine-grained visual features. These features are then aggregated and integrated into the FFN computation, with the reinforcement strength adjusted according to their regional contributions. Experiments on several LVLMs and benchmarks show improvements in hallucination mitigation, and ablation studies support the effectiveness of the individual components.

Critical strengths
  1. I find the focus on visual access during decoding interesting. In particular, using contrastive trajectories to characterize the effect of visual-token access, rather than relying solely on attention scores, offers a potentially useful perspective. The region-level retrieval and re-encoding also allow the method to work with more localized visual evidence instead of simply reinforcing individual patches.

  2. The experimental results are quite encouraging, with consistent improvements on several hallucination benchmarks. The authors also include reasonable ablations for the main components, which is appreciated.

Critical weaknesses
  1. My main concern is that the motivation behind several design choices is not sufficiently convincing. The method involves quite a few steps and heuristics, from contrastive layer selection to visual retrieval and FFN reinforcement. For example, it is not immediately clear why a decrease in the contrastive trajectory should indicate a loss of useful visual evidence. Although the ablations show that these components help empirically, I still find the overall design somewhat difficult to justify.

  2. I am also not fully convinced by the representation matching and injection process. The method directly computes similarities between decoder hidden states and visual tokens, and then integrates the retrieved features into the FFN output. I would like to see more analysis supporting the compatibility of these representations and the actual effects of the intervention. For example, some visualization or analysis of the similarity scores, retrieved regions, and resulting hidden-state changes would make the method much more intuitive and convincing.

  3. The method seems computationally expensive, especially considering the dual-branch decoding and repeated region re-encoding. However, I could not find an efficiency analysis in the paper. This is an important omission for an inference-time method.

  4. The backbones are too old, the paper lacks evalution on advanced LVLMs such as Qwen3VL/Qwen3.5.

Questions for authors
  1. Could the authors provide more intuition or preliminary evidence for the key design choices, especially the contrastive trajectory criterion, similarity-based retrieval, and FFN injection? Are there any analyses showing that these operations behave as intended?

  2. Could the authors provide an efficiency comparison with the baselines, including inference latency and the overhead of region re-encoding?

  3. Do the authors plan to release the code?

Reviewer_SN59OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VisDelta, a training-free method based on controlled visual-access contrast and visual evidence enforcement for mitigating object hallucination in large vision-language models (LVLMs). The method is evaluated across several LVLM backbones, including LLaVA-1.5-7B, InstructBLIP, Qwen-VL, and GLM-4V-9B. The reported results demonstrate reductions in object hallucination on CHAIR and POPE; additional evaluations on MME and MMBench assess general multimodal capabilities.

Critical strengths
  1. The proposed method can be applied to existing LVLMs to mitigate object hallucination without additional training. In addition, the combination of contrastive magnitude change and entropy is interesting.
  2. The method is evaluated across multiple LVLM backbones and benchmarks, providing evidence of its effectiveness across the tested settings.
  3. The paper is generally well organized and easy to understand.
Critical weaknesses
  1. Eq. (2) measures the normalized difference between the last-token attention outputs of the standard and visually blocked branches. However, the explanation that the two decreases determine whether access to visual tokens is blocked is overly heuristic; the causal relationship lacks theoretical or experimental support.
  2. The sample-dependent selection of intervention layers may complicate batched inference, as different samples may require additional operations at different depths. In addition, DCT introduces dual-branch computation, whereas CFER requires additional crop encoding and feature integration. I would like the authors to provide the inference latency relative to the base model and the overhead of the other methods.
  3. Although the paper evaluates many models and considers fair comparisons, these models are too outdated for 2026. The aim should be to include comparisons with more recent fundamental LVLM models, which would make a greater contribution and be more helpful to the community.
  4. The role of the scaling factor γℓint\gamma_{\ell_{\mathrm{int}}} in Eqs. (7)–(10) is unclear. It cancels from the concentration score in Eq. (9) when ϵ\epsilon is neglected. Furthermore, in Eq. 10, both M~\widetilde{\mathbf{M}} and μ(M~)\mu(\widetilde{\mathbf{M}}) will also be simplified. The authors need to re-examine this part of the design.
Questions for authors
  1. I don't understand the calculation of η\eta in Eq. 3. Shouldn't this theoretical value be 0? Please clarify the computational differences that produce this discrepancy, report its typical magnitude relative to the observed changes in rℓr_\ell, and explain whether it materially affects layer selection if it's 0.

  2. Although α2\alpha_2 is described as an additive baseline in the CCR interpolation coefficient, its experimental value is not reported. Moreover, Table 4 varies α1\alpha_1 from 0.18 to 0.09, whereas Appendix D refers to αmax⁡=0.18\alpha_{\max}=0.18. Please specify the default values of α1\alpha_1 and α2\alpha_2, define the relationship between αmax⁡\alpha_{\max}.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.