acceptodds
Withdrawn from ICLR 2027

DriftDPO: Mechanism-Guided Preference Optimization for Hallucination Mitigation in Deep-Fusion Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Object hallucination in vision-language models (VLMs) is widely attributed to language-prior dominance, motivating contrastive decoding (CD) methods that suppress this prior. However, CD consistently worsens hallucination on modern deep-fusion VLMs, suggesting the prevailing theory does not hold for these architectures. Through token-level probing of internal representations, we discover that the theory is not merely incomplete but inverted: on deep-fusion VLMs, hallucination is driven by visual-mean alignment rather than language-prior dominance, and this mechanism is architecture-gated—active in deep-fusion models, absent in shallow-projection models, and reversed in next-generation ones. Based on this finding, we propose DriftDPO, which derives its preference signal entirely from the model's own hidden states, requiring no external annotation. A per-token analysis reveals that the caption-level signal is biased by function-word dominance, necessitating a direction correction step to align the training objective with the object-level mechanism. DriftDPO achieves state-of-the-art hallucination reduction on multiple benchmarks while increasing coverage at zero inference overhead—the only method to simultaneously achieve all three. Mechanism verification confirms that training genuinely internalizes the signal into model weights rather than superficially suppressing it, revealing a coupling property unique to self-supervised rewards.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language models,Object hallucination,Contrastive decoding,Visual-mean alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ydfHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DriftDPO, which ranks model-generated image captions using the alignment between generated-token and visual-token representations, then trains on the resulting preference pairs with DPO. It reports reduced object hallucination and increased object coverage without additional inference steps.

Critical strengths

None

Critical weaknesses
  1. The central mechanism claim is unsupported. An AUC of 0.589 shows limited association, while selecting captions by cosine similarity does not establish causality. The paper also conflates two different comparisons—hallucinated objects versus background tokens, and hallucinated versus correct objects—to explain its preference reversal. The proposed architectural explanation lacks controlled evidence.

  2. The experimental evidence falls far short of the claims. The main text provides verifiable training results primarily for one 3B model, while referring to an appendix absent from the provided PDF for cross-model results and key ablations. Moreover, CHAIRi falls as object coverage rises, but the paper does not report the raw counts needed to establish that the absolute number of hallucinated object mentions decreases.

  3. Novelty and baselines are inadequate. Prior work already studies model-generated preference pairs for hallucination mitigation. The paper neither establishes a clear advance over such methods nor compares against relevant training-based baselines, making its “first,” “only,” and “state-of-the-art” claims unconvincing.

Questions for authors

See weakness

Reviewer_Rqs1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the phenomenon that contrastive decoding exacerbates object hallucinations in some VLMs. The authors propose constructing preferences based on the cosine similarity between generated token hidden states and the mean of visual token representations, followed by direction correction and LoRA-DPO training. The main results table reports a reduction in CHAIRi from 0.1810 to 0.1408, along with an increase in object mentions, without requiring an additional scoring branch at inference time.

Critical strengths
  1. The method uses an internal scoring signal to construct offline preferences and still employs standard greedy decoding after training. In the main results table, both CHAIRi and CHAIRs outperform the original model, while the number of object mentions increases. This provides attractive empirical results for hallucination mitigation without additional inference branches.

  2. This experimental comparison suggests that when internal statistics are converted into preferences, the ranking direction can significantly affect output behavior. Even if the mechanistic explanation remains to be validated, this observation is still worth investigating.

Critical weaknesses
  1. The so-called "causal validation" is actually a reranking of existing candidates based on cosim. This experiment can support the claim that "the signal has value for selection," but it does not directly intervene on hidden states under a fixed image and generation prefix, nor does it rule out factors such as length and object count. Therefore, it cannot yet be confirmed that cosim is a cause of hallucination rather than a metric correlated with output characteristics.

  2. The architectural attribution also requires a more rigorous definition. LLaVA-1.5, Qwen2.5-VL, and InternVL3.5 all contain visual feature projection or MLP connection modules. Although these models are not identical, "having a lightweight projection" and "performing fusion within a shared Transformer" are not inherently mutually exclusive categories. The current cross-model comparisons do not isolate factors such as fusion depth, training data, and language backbones, and are therefore insufficient to attribute the differences to fusion depth.

  3. The current comparisons are mainly with inference-time methods and have not yet established the independent advantages of cosim-based preference construction over other self-supervised preference approaches (e.g., SeVa [1]).

[1] Zhu K, Zhao L, Ge Z, et al. Self-supervised visual preference alignment[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 291-300.

Questions for authors
  1. I think the most important issue that needs to be addressed is that the claim about the causal mechanism is not yet sufficiently supported. The existing experiments show that cosim can be used for candidate ranking and preference construction, but Section 3.5 intervenes on the output selection strategy, while DPO simultaneously modifies multiple model components. Neither isolates the specific role of visual-mean alignment in hallucination generation. Therefore, the effectiveness of the method cannot directly serve as evidence for the causality of the mechanism. The authors need to provide additional controlled interventions or limit their conclusion to cosim being an effective internal proxy signal.

  2. Please provide operational definitions of shallow-projection and deep-fusion, as well as the actual differences in fusion locations and depths across the models. Can the effects of language backbones, training data, and model scale be ruled out? If controlled experiments are not yet available, I suggest limiting the conclusion to "differences observed across the evaluated model families."

  3. Please include comparisons with existing comparable self-supervised post-training methods, rather than only comparing with inference-time methods.

Reviewer_WoKoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies object hallucination in deep-fusion vision-language models such as Qwen2.5-VL and InternVL3.5. The authors observe that contrastive decoding methods (VCD, ICD, M3ID), which are designed to suppress the language prior, make hallucination worse on these models. By probing internal representations, they argue that the language-prior explanation does not hold for deep-fusion models and that hallucination is instead linked to "visual-mean alignment", the cosine similarity between a generated token's hidden state and the mean of the visual token states. They also claim this behaviour depends on the architecture: it is present in deep-fusion models, absent in LLaVA-1.5, and reversed in Qwen3-VL. Based on this, they propose DriftDPO, which samples several captions per image, scores each by its average alignment, and trains a LoRA adapter with DPO on the highest- and lowest-scoring pairs, with no external annotation. A per-token analysis shows that the caption-level score is dominated by function words, so the authors add a "direction correction" step that reverses which caption is preferred. On Qwen2.5-VL-3B, DriftDPO reduces COCO CHAIRi by 22.2% while increasing the number of mentioned objects, with no inference overhead, and it also improves POPE, MMHal-Bench and AMBER. Further analyses look at how the signal changes after training and why more preference data hurts performance.

Critical strengths
  1. A useful empirical observation. Contrastive decoding increases CHAIRi on Qwen2.5-VL-3B by 13–25% (Table 2) while reducing it on LLaVA-1.5. If this holds more broadly, practitioners should know that methods validated on LLaVA-1.5 may not carry over to newer architectures.

  2. Annotation-free training with good results at no inference cost. On Qwen2.5-VL-3B, DriftDPO reduces CHAIRi from 0.1810 to 0.1408 and increases coverage from 6.7 to 8.66 (Table 2), using a self-generated preference signal and standard greedy decoding. It also improves POPE (+3.4 accuracy) and MMHal-Bench (Sec. 5.4). Training is cheap (about 2 hours on one L40s).

Critical weaknesses
  1. The paper's own analysis contradicts its main mechanism claim.

    • The AUC of 0.589 in Table 1(b) compares object tokens against background tokens (Sec. 3.2), not hallucinated objects against correct ones. It shows that object tokens have higher alignment than function words, which says nothing about hallucination.
    • When hallucinated and correct objects are compared directly, the AUC is 0.426 (Sec. 3.4): correct objects have higher alignment than hallucinated ones. This is the opposite of the claimed mechanism.
    • The final method therefore prefers high-alignment captions (Sec. 4.3). This fits the ordinary view that stronger visual grounding means less hallucination, not an "inverted" theory.
    • Similarly, the language-prior AUC of 0.439 compares objects with function words, which naturally have different prior probabilities. It does not test the language-prior theory, which is about hallucinated versus correct objects.
  2. The causal and gating claims are inconsistent.

    • Picking the lowest-alignment caption out of 4 reduces CHAIRi by 12% (Sec. 3.5), but the final DPO method chooses the highest-alignment caption.
    • Choosing among sampled captions is not an intervention on the signal, so it doesn't show causality.
    • Sec. 3.3 says DriftDPO applies when AUC > 0.5. Sec. 4.3 then says Qwen2.5-VL and InternVL3.5 have "AUC < 0.5", although Table 1(c) lists them at 0.589 and 0.618. Here the caption-level and per-token AUCs are being mixed.
    • Sec. 5.3 says "all four deep-fusion models (AUC > 0.5) worsen under CD", but Qwen3-VL-4B is listed as deep-fusion with AUC 0.390.
    • The claim rests on one shallow-projection model and a few deep-fusion models, with AUC values close to 0.5 (0.39–0.62).
Questions for authors
  1. Table 1(b) compares object tokens with background tokens. What are the AUCs of ppriorp_{prior}, cosim and Δ\Deltalogit for hallucinated vs. correct objects only? Given that this AUC is 0.426 for cosim (Sec. 3.4), how do you reconcile it with the claim that high alignment drives hallucination?
  2. CoSim-Beam selects the lowest-alignment caption and reduces CHAIRi (Sec. 3.5), while DriftDPO prefers the highest-alignment caption. Which direction is correct, and why do they disagree?
  3. How does DriftDPO compare with DPO using pairs chosen by caption length, number of objects, or CLIP score? Do max-alignment captions tend to be longer?
  4. Which images were used to build the preference data, and how was the number of pairs selected (Sec. 5.7)? Was it tuned on the COCO val2017 images used for evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.