acceptodds
Under review as a conference paper at ICLR 2027

TAVCD: TARGET-AWARE VISUAL CONTRASTIVE DECODING FOR HALLUCINATION MITIGATION IN VISION- LANGUAGE MODELS

Abstract

Visual Contrastive Decoding (VCD) mitigates object hallucination in Large Vision-Language Models (LVLMs) by contrasting positive-branch logits (original image) against a negative branch (degraded image), weighted by a fixed scalar α. We expose a fundamental limitation of this paradigm, formalised as the Contrastive Correctability Condition (C³): no fixed positive α can correct a hallucinated token h whenever h simultaneously dominates the ground-truth token g in both the baseline logit distribution and the visual contrastive signal D(v)=ℓ⁺(v)−ℓ⁻(v). We further prove a universal insufficiency result: for any fixed α₀∈ℝ there exist realizable decoding configurations under which α₀ either fails to correct or actively amplifies hallucination. Empirically, across several backbones on MSCOCO, C³ is violated at approximately 26% of object-generating steps, confirming that fixed-α VCD is counterproductive at roughly one in four critical moments. Building on this theory, we propose TAVCD, a lightweight per-step adaptive contrastive decoder. At each decoding step TAVCD builds a compact 10-dimensional feature vector from observable logit statistics and routes it through a shared-trunk gate+magnitude MLP (≈12K parameters). The gate decides whether contrastive intervention is warranted; the magnitude head selects the signed strength from a fine-grained 40-point α grid. TAVCD is trained offline on logged decoding traces with ground-truth onset supervision and adds negligible wall-clock overhead at inference. On MSCOCO and PASCAL VOC captioning across five diverse LVLMs, TAVCD reduces hallucination on the majority of backbones, achieving up to 24.0 pp CHAIRᵢ improvement over standard decoding and up to 5.8 pp CHAIRₛ improvement over fixed-α VCD.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
hallucination mitigation, visual contrastive decoding, vision-language models, adaptive decoding, object hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aekxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper characterizes when a contrastive coefficient can promote a ground-truth object token above hallucinated competitors. It identifies configurations that no positive coefficient can correct and proposes TAVCD, a small controller that predicts a signed coefficient from logit statistics at each decoding step. The controller is trained on oracle labels obtained from logged VCD trajectories. Experiments cover five frozen vision-language models on MSCOCO and PASCAL VOC captioning.

Critical strengths

The distinction between pairwise blocking and incompatible promotion intervals is useful. Although the analysis follows directly from an affine logit margin, it explains why increasing a positive coefficient can worsen a particular error and motivates considering negative coefficients. The experiments consistently improve or match VCD on the reported hallucination metrics across both datasets, with no updates to the backbone.

Critical weaknesses
  1. The experiments do not yet establish the value of the learned controller over simpler adaptation. On MSCOCO, the CHAIRi reductions relative to VCD are 1.0, 0.1, 1.3, 0.3 and 0.1 percentage points in Table 1's backbone order. These are single-seed results on 500 images, without uncertainty estimates. The paper does tune a global positive coefficient on validation data, but that does not replace a comparison with a simple gate using one tuned signed coefficient. Such a control would distinguish the value of selecting when to intervene from that of learning a 40-bin magnitude distribution. The headline 24-point improvement on LLaVA VOC is also achieved by VCD: both reach CHAIRi 14.0. It is a valid comparison with Regular decoding, but does not establish the added value of TAVCD.

  2. The deployment features need clarification. Table 2 defines t/T using the total caption length, which is unavailable before generation terminates. If inference substitutes a maximum generation length or an estimate, the paper should specify it and use the same definition when training the predictor. As written, the online feature computation cannot be reproduced unambiguously.

  3. The oracle labels provide a weaker guarantee than the term “structurally uncorrectable” suggests. Appendix B searches a bounded, discrete grid on the stored top-50 support; failure there does not establish that the full real-valued feasible set is empty. A C3 violation excludes positive coefficients, not necessarily negative ones.

  4. The recall tradeoff is substantial for BLIP-2 on VOC: 61.6 with VCD falls to 49.0 with TAVCD. CHAIRi improves from 16.4 to 16.2 and CHAIRs from 18.0 to 14.2, but these reported gains accompany considerably less object coverage. Object-mention counts and caption lengths would help characterize this tradeoff.

  5. The reported “less than 1%” overhead concerns the head, not the two-branch decoder. End-to-end latency against Regular decoding and VCD is still missing.

Questions for authors

What value replaces the eventual caption length T during inference, and was that same definition used to construct the training features?

Reviewer_WdqFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes TAVCD, which trains a lightweight controller to adjust the strength and direction of visual contrastive decoding at each token to reduce object hallucinations. It evaluates the method on two image captioning datasets across five VLM backbones.

Critical strengths

The method is evaluated across five VLM backbones, providing reasonably broad validation, although several models are outdated.

Critical weaknesses
  1. Limited novelty: Prior work, including Octopus and CATCH, already uses token-level adaptive contrastive decoding. The paper should compare directly with them.

  2. Outdated backbones: Evaluation should include more recent and stronger VLMs.

  3. Modest gains: Improvements over fixed-coefficient VCD are often small, making the practical benefit unclear.

Questions for authors

See weakness

Reviewer_GXWrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem that Visual Contrastive Decoding (VCD) with a fixed coefficient may fail to correct or even amplify object hallucinations. Through single-step logit geometry analysis, the authors characterize the conditions for correctability and propose TAVCD, which uses a lightweight gating network to predict a signed contrastive coefficient at each decoding step based on observable statistics. The CHAIR metrics show overall improvements across five backbones and two image captioning datasets.

Critical strengths
  1. The paper formulates the competition between correct tokens and hallucinated tokens as linear inequalities involving α\alpha, distinguishing between two cases: a single competing token blocking correction and conflicts among multiple feasible intervals. This clearly explains why continuously increasing a positive coefficient is not necessarily effective and provides a reasonable motivation for allowing intervention in the opposite direction.

  2. The method freezes the LVLM and trains only a controller with approximately 12K parameters, and is validated on five backbones. In Table 1, compared with fixed-coefficient VCD, TAVCD improves or matches both CHAIR metrics across all backbone–dataset combinations.

Critical weaknesses
  1. The paper generally characterizes related methods as using fixed, global contrastive coefficients, but this positioning is not sufficiently comprehensive. M3ID[1] has already adopted confidence-based gating and intervention strengths that vary with generation steps, while MoD[2] dynamically selects complementary or contrastive strategies based on the consistency between branch outputs.

  2. Some results show a significant loss in recall. For example, on VOC with BLIP-2, compared with VCD, CHAIRs decreases from 18.0 to 14.2, but CHAIRi only decreases from 16.4 to 16.2, while Recall drops from 61.6 to 49.0. This suggests that the sentence-level improvement in this setting may partly result from mentioning fewer objects, rather than more accurately correcting hallucinations. Furthermore, the maximum 24-percentage-point improvement in CHAIRi highlighted in the abstract corresponds to LLaVA-1.5 on VOC, where CHAIRi decreases from 38.0 to 14.0. However, fixed-coefficient VCD already achieves the same value. This does not invalidate the positive results in other settings, but it limits the strength of the overall conclusions.

[1] Favero A, Zancato L, Trager M, et al. Multi-modal hallucination control by visual information grounding[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 14303-14312.

[2] Chen X, Zhang Y, Liu Q, et al. Mixture of decoding: An attention-inspired adaptive decoding strategy to mitigate hallucinations in large vision-language models[C]//Findings of the Association for Computational Linguistics: ACL 2025. 2025: 8525-8542.

Questions for authors
  1. Why does this paper primarily use relatively older VLMs rather than more recent models?

  2. Could the authors provide comparisons with existing adaptive decoding methods such as M3ID and MoD under the same experimental settings, and conduct ablation studies on "gating only + fixed positive coefficient," "non-negative adaptive coefficient," "signed coefficient without gating," and the full method?

  3. Given the significant drop in Recall on BLIP-2 VOC, could the authors report the average caption length, the number of correct and incorrect object mentions, and confidence intervals computed from paired comparisons at the image level? Does TAVCD still outperform VCD at comparable caption lengths or recall levels? Alternatively, a supplementary human evaluation jointly assessing factual correctness and completeness could provide additional supporting evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.