acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Contrastive Decoding: Evidence Guided Branch Assignment for Multimodal Hallucination Mitigation

Abstract

Multimodal large language models (MLLMs) integrate visual perception with the language understanding and generation capabilities of large language models, enabling a wide range of multimodal tasks. However, object hallucination remains a critical challenge that undermines the reliability of MLLMs. Existing contrastive decoding methods mitigate hallucinations by constructing positive and negative branches through visual perturbation, token selection, or attention modulation. However, the roles of these branches are often determined by predefined operations rather than by explicitly verifying whether their contents are supported by the input image. To address this limitation, we propose Visual Evidence guided Role Assignment (VERA), an inference time hallucination mitigation framework that verifies intermediate visual hypotheses before constructing contrastive branches. VERA assigns visually supported components to the positive branch, while uncertain hypotheses are reverified before their unresolved residuals are used for negative suppression. To prevent residual information from being inadvertently converted into positive evidence during contrastive subtraction, VERA further constrains the residual branch to provide suppression only. Extensive experiments on CHAIR, POPE, and five general visual question answering benchmarks demonstrate that VERA effectively reduces object hallucinations while preserving the general multimodal understanding capabilities of MLLMs across LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal large language models, object hallucination, contrastive decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vz2fOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the method to construct contrastive branches in contrastive decoding for MLLM hallucination mitigation. The authors argue that exsiting strategies construct negative branches with predefined operations. To address this limitation, the authors propose Visual Evidence guided Role Assignment (VERA), an inference-time hallucination mitigation method which verifies intermediate visual hypothese before constructing contrastive branches. Experiments on CHAIR, POPE, and five general visual question answering benchmarks show the effectiveness of the proposed method.

Critical strengths
  1. On the whole the writing is clear, and the core of the idea can be grasped.
  2. Experiments on several benchmarks show the improvement by the proposed method.
Critical weaknesses
  1. Although the authors report the results of the proposed method on several benchmarks, the evaluation is neither enough nor fair. In table 1, the authors report results on CHAIR and POPE. However, 500 random samples from COCO 2014 validation set is far from enough: The variance is very high; the performance gain, such as an improvement of 0.87% on accuracy for LLaVA V1.5 on POPE adversarial set (equivalent to 4 more correct answers within 500 testcases), do not really imply substantial improvement. To make things worse, as reported by HalluEdit, the performance of HalluEdit for LLaVA V1.5 on POPE adversarial set is 82.5, rather than the 80.13 reported by the authors. There are strong possibilities that the authors suppress the performance of the baselines.
  2. The authors did not clearly define the meaning of some symbols, such as M+\mathcal{M}^{+} and M−\mathcal{M}^{-}.
Questions for authors
  1. Please provide the experiment results under wider and fairer evaluation protocol.
  2. What are the meanings of {h_1, h_2, \cdots, h_U\}? How are they computed?
  3. What are the meanings of M+M^{+} and M−\mathcal{M}^{-}?
Reviewer_spyzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VERA, a framework for mitigating multimodal hallucinations through evidence-guided contrastive branch construction. It generates intermediate hypotheses, estimates their visual support using a learned verifier, and constructs supported and residual memories for contrastive decoding. A one-sided residual constraint is introduced to prevent residual-induced positive logit contributions.

Critical strengths
  1. The paper addresses a relevant limitation of contrastive decoding: suppressing hallucinations can also suppress valid visual information.
  2. The experiments include multiple backbones, component ablations, and measurements of inference latency and memory consumption.
Critical weaknesses
  1. Experiments are restricted to LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, all at the 7B scale. These models provide continuity with earlier work, but the absence of newer architectures and larger models substantially limits the relevance of the conclusions. Newer families such as Qwen3-VL have been available since 2025. It remains unclear whether the proposed verifier and branch construction remain effective when the underlying model has stronger visual grounding and different hallucination patterns.
  2. The reported full-method results are inconsistent across experimental sections without an adequate explanation. On Qwen2.5-VL, Table 1 reports VERA’s CHAIRs/CHAIRi as 15.00/7.31, whereas the full-method row in Table 2 reports 12.00/5.99. The main POPE results yield an average F1 of approximately 86.70, compared with 84.72 in Table 2. Figure 5(a) and the default configuration in Table 13 report still different F1 values. Different subsets or configurations could explain these discrepancies, but the manuscript does not clearly identify them. This prevents the reader from reliably connecting the ablation results to the main comparison.
  3. The computational overhead is substantial, while the advantage over strong baselines is not consistently compelling. Figure 5(b) reports 4.69 seconds per sample, or 2.78× vanilla latency, making VERA the slowest method in the comparison. Describing this as a “modest” inference delay understates the practical cost. Meanwhile, some gains over competing methods are very small: on LLaVA-NeXT, average POPE F1 is 89.57 versus 89.34 for PND and HulluEdit, and VERA’s CHAIRi is worse than both SID and CausalLens. Without uncertainty estimates and comparisons at matched inference budgets, the claimed favorable deployment trade-off remains insufficiently established.
Questions for authors
  1. Can the authors evaluate VERA on a newer model family and a larger backbone to establish whether its benefits extend beyond the three older 7B models?
  2. What differences in evaluation subsets or configurations explain the full-method results in Tables 1, 2, and 13 and Figure 5(a)? Can these settings and results be reconciled explicitly?
  3. Are the small gains over strong baselines statistically reliable, and does VERA remain advantageous when methods receive comparable inference-time budgets?
Reviewer_9fKLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VERA constructs contrastive decoding branches from generated visual hypotheses. A learned support estimator compares their contextual states with representations retrieved from the original visual keys and values. A second lookup refines support, after which supported and residual memories are injected into separate branches. A rectified residual-logit correction prevents a direct positive logit contribution from residual memory. For binary existence questions, an additional support-based calibration boosts “yes” tokens. The backbone is frozen; the Qwen2.5-VL verifier has roughly 22.1M parameters and is trained on 10,000 COCO images. Results cover three 7B backbones, CHAIR, POPE, general VQA, and several diagnostics.

Critical strengths
  1. The proposal goes beyond assuming that a visual perturbation defines a valid negative branch. It learns support from image annotations and uses that score to construct complementary memories.

  2. The empirical evaluation is broad and includes substantial improvements. On Qwen2.5-VL, CHAIRs decreases from 35.66 to 15.00 and POPE average accuracy increases from 81.33 to 87.88, with similar reported output lengths (Tables 4–5). Gains also appear on both LLaVA backbones.

Critical weaknesses
  1. The main results and supporting analyses lack a consistent, traceable evaluation configuration. For Qwen2.5-VL, Table 1 reports full VERA at CHAIRs/CHAIRi = 15.00/7.31, whereas Table 2 reports 12.00/5.99. Table 5 gives POPE average accuracy/F1 = 87.88/86.70, while the full method in Figure 5(a) reaches 89.53/88.85. These may reflect legitimate subsets or settings, but the manuscript does not identify them. Furthermore, Appendix D.2 specifies 500 CHAIR images: a single caption-level CHAIRs estimate would then change in increments of 0.2 percentage points, which does not explain values such as 35.66 or 25.33 without additional averaging or a different denominator. These ambiguities make it difficult to connect the component evidence to the headline comparison or reproduce the claimed improvements.

  2. The POPE gains do not isolate branch assignment from direct answer calibration. Appendix C.6 adds a support-dependent bonus to “yes” logits, using a verifier supervised on COCO object presence/absence (Appendix D.4). This provides an alternative route to recovering positive answers beyond the proposed memory-based contrast. Table 13 shows that changing this calibration coefficient affects F1, but does not include a zero-calibration setting; Table 2 likewise lacks a calibration-only baseline. Existing branch ablations are useful, and the CHAIR gains cannot be explained by binary-answer calibration. Nevertheless, the evidence does not establish how much of the central POPE recovery comes from the proposed branch mechanism rather than the learned existence predictor and its direct logit adjustment.

Questions for authors
  1. Clarification: Please specify the sample IDs/counts, checkpoint, decoding settings, and aggregation for Tables 1–2, Figure 5(a), and Table 13, and reconcile the CHAIR denominator with the stated 500-image protocol. Do the conclusions hold when these comparisons use the same configuration?
  2. Targeted comparison: Can you compare full VERA, VERA with claim calibration disabled, and the same verifier/calibration applied without memory injection on matched POPE examples? Please also specify the actual visual perturbation and attention configuration of the “base residual branch” in Appendix C.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.