acceptodds
Under review as a conference paper at ICLR 2027

CAVCD: Candidate-Aware Visual Counterfactual Decoding for Mitigating Object Hallucinations in Large Vision-Language Models

Abstract

Large vision-language models (LVLMs) suffer from object hallucination, where generated responses mention objects absent from the input image. Contrastive decoding alleviates this problem by contrasting original and perturbed predictions, but offers limited control over which candidate tokens to correct and how strongly, potentially disrupting otherwise correct predictions. To address these limitations, we propose Candidate-Aware Visual Counterfactual Decoding (CAVCD), a training-free method built on the idea that intervention-induced prediction changes should be assessed for individual candidates before guiding token selection. CAVCD first adapts visual interventions to the generation process, restricting visual access at the initial step and applying softer attention attenuation during subsequent generation. During continuation, it evaluates candidate responses across decoder layers and combines selected responses into a target distribution that guides logit correction. An adaptive gate then regulates correction strength, limiting excessive updates that could override image-supported token choices. Together, these designs make correction more selective and adaptive, helping suppress hallucinations while preserving plausible predictions. Extensive experiments demonstrate that CAVCD effectively mitigates object hallucination across multiple evaluation metrics.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Vision-Language Models, Hallucination Mitigation, Contrastive Decoding, Training-Free Inference, Trustworthy AI

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_6SzCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Candidate-Aware Visual Counterfactual Decoding (CAVCD), which employs attention-guided visual interventions to construct a hallucination distribution and adaptively regulate the strength of contrastive decoding. Experiments on the POPE and CHAIR benchmarks demonstrate that CAVCD can effectively mitigate hallucinations in large vision-language models to some extent.

Critical strengths

None

Critical weaknesses
  1. The novelty of this work is limited. Contrastive decoding is a relatively mature research direction, and numerous studies have explored hallucination mitigation through input perturbations or attention interventions. In particular, methods such as SID[1] and ONLY[2] have already investigated contrastive decoding by selectively suppressing or enhancing specific modalities (visual or textual tokens). Compared with these methods, the proposed approach mainly introduces modifications to the intervention and correction strategies, without demonstrating substantial methodological novelty or providing sufficiently compelling motivation to distinguish itself from existing work.
  2. The experimental evaluation is not sufficiently convincing. The backbone models used in this study, such as LLaVA-1.5 and InstructBLIP, are relatively outdated, and the baselines do not adequately cover recent advances in this field. These limitations weaken the persuasiveness of the experimental results. Furthermore, I noticed that some baseline results in Table 1 differ from those reported in their original papers, and several baselines even underperform the original models in certain settings. The authors should clarify the sources of these results and provide details of the reproduction settings to ensure fair and reliable comparisons.

[1] ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models. [2] Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models.

Questions for authors

no other questions

Reviewer_gsBROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CAVCD (Candidate-Aware Visual Counterfactual Decoding), a training-free decoding method for alleviating object hallucinations in large vision-language models (LVLMs). Existing contrastive decoding methods usually correct predictions by directly utilising the logits difference between the normal branch and the negative branch, but this operation may suppress valid visual evidence, leading to unnecessary prediction changes. CAVCD consists of three main components. First, Hard-to-Soft Visual Intervention retains only the visual token with the lowest attention in the first decoding step and attenuates the visual token with the highest attention in subsequent steps. Second, Candidate-Aware Target Construction compares the prediction changes between the normal branch and the negative branch across multiple decoding layers, constructing a candidate target distribution using the cosine similarity between the loss gradient direction of candidate tokens and the log-probability shift caused by interventions. Finally, Gated Token Correction adaptively adjusts the strength of logits correction based on the concentration of the target distribution and the Jensen–Shannon divergence between the normal branch and the negative branch. Experiments were conducted on LLaVA-1.5-7B and InstructBLIP, using CHAIR to evaluate object hallucinations in open-ended image descriptions and POPE to assess object existence judgements. Results show that the method outperforms existing training-free decoding baselines on some metrics, and ablation studies analyse the effects of different components.

Critical strengths
  1. The problem has practical significance, and the overall framework design is relatively complete. The paper focuses on the issue that contrastive decoding may mistakenly penalise correct predictions and attempts to address this problem through candidate-level evaluation, staged visual intervention, and adaptive correction. The overall design has a clear process, with each module aimed at reducing hallucinations and avoiding overcorrection.
  2. The experiments cover various models and evaluation settings. The paper evaluates open-ended image descriptions and object presence judgments on two LVLMs and compares multiple zero-shot decoding baselines. On LLaVA-1.5-7B, CHAIRs decreased from 46.0% to 38.4%, and CHAIRi from 12.2% to 10.1%; similarly, noticeable improvements were achieved on both metrics for InstructBLIP. The paper also provides multiple ablation and parameter sensitivity experiments, offering some basis for analysing method behaviour.
Critical weaknesses
  1. The effectiveness of counterfactual visual interventions has not been fully validated. Since visual masking is applied only after certain layers of the model, earlier layers have already processed the complete visual input, and information from masked tokens may have propagated to other hidden states. Therefore, prediction differences between the normal and negative branches may reflect attention redistribution or changes in feature representations rather than the effects of removing specific visual evidence. The paper lacks direct experiments to verify whether the intervention effectively isolates the target visual information and whether the resulting prediction differences reliably measure the model's dependence on visual information.

  2. The effectiveness of the gradient alignment mechanism is insufficiently supported. The cosine similarity between intervention-induced log-probability changes and cross-entropy gradients does not necessarily reflect the degree of visual evidence supporting candidate content or its actual correctness. Moreover, Table 3 shows that gradient alignment reduces CHAIRs only from 38.8% to 38.4% compared with directly using log-probability differences. This marginal improvement lacks statistical significance analysis, and the paper does not directly evaluate the relationship between alignment scores and candidate correctness. Therefore, the independent contribution of this mechanism remains unclear.

  3. The preservation of visual information is insufficiently evaluated. Attenuating high-attention visual tokens may suppress not only hallucination-related information but also visual evidence necessary for correct predictions. Although the paper reports hallucination metrics and generation statistics, it lacks direct evaluations of correct object mentions, attribute recognition, and relationship descriptions. Therefore, it remains unclear whether hallucination reduction comes at the expense of the completeness and accuracy of visual descriptions.

  4. The rationale for adaptive gating is insufficiently justified. Combining the two Jensen–Shannon divergence signals with equal weights lacks sufficient theoretical justification, and the paper does not directly evaluate whether the gating coefficient can effectively identify predictions that require correction. Appendix Table 6 shows that adaptive gating reduces CHAIRs by only 0.6 percentage points compared with a fixed gating coefficient (38.4% vs. 39.0%), without statistical significance analysis or validation across different models and tasks. Therefore, the necessity of adaptive gating has not been sufficiently established.

  5. The experimental scope is limited, and the additional inference overhead is substantial. Experiments are limited to LLaVA-1.5-7B and InstructBLIP, while the CHAIR evaluation uses only 500 randomly selected COCO images, without repeated sampling experiments or confidence interval analysis. Moreover, CAVCD requires 9.359 seconds per image, approximately 2.19 times the inference time of the original decoding method and about 10% more than SID. Given the inconsistencies in runtime implementations across methods, the actual computational overhead introduced by the algorithm and its accuracy–efficiency trade-off require further investigation.

Questions for authors
  1. On the effectiveness of counterfactual visual interventions. Since the network layers before the intervention layer can still access all visual tokens, some visual information might have already propagated to the hidden states of other tokens through earlier layers. How do the authors show that the prediction differences between the normal branch and the negative branch effectively reflect changes in the model's reliance on visual evidence, rather than mainly resulting from attention redistribution or visual information already propagated? Could you provide direct experimental analysis to support this explanation?
  2. On the necessity of the gradient alignment mechanism. Table 3 shows that compared to the simpler candidate-level log-probability difference, the gradient alignment mechanism only brings a 0.4 percentage point improvement in CHAIRs. Could the authors provide paired statistical tests or confidence intervals to verify the statistical reliability of this improvement? Also, what specific information does gradient alignment capture that simple probability differences cannot reflect? Is there any evidence showing that the alignment score correlates with the correctness of candidate tokens or visual grounding
  3. On retaining correct visual predictions. How do the authors verify that attenuating high-attention visual tokens can selectively suppress hallucinations without simultaneously weakening the important visual evidence that underpins correct predictions? Could the impact of this method on correct object recognition, attribute judgement, and relationship description be further analysed to support the core claim of avoiding over-correction while preserving correct predictions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.