acceptodds
Withdrawn from ICLR 2027

Dual-View Anchored Decoding for Mitigating Object Hallucination in MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) often suffer from object hallucination, generating objects or attributes that are absent from the visual input. We attribute this failure to the interaction between misleading visual evidence and grounding attenuation: some visual tokens induce uncertain or misleading textual interpretations, while autoregressive decoding progressively amplifies weakly grounded continuations as the textual context grows. Crucially, high epistemic uncertainty does not necessarily imply useless visual information, since a visual token may cover both misleading cues and useful object-specific details. We propose Dual-View Anchored Decoding (DVAD), a training-free decoding strategy that preserves the original full view and constructs an auxiliary anchor view by selectively suppressing visual tokens likely to carry unstable evidence. DVAD jointly decodes the two views with a shared generated prefix and fuses their next-token distributions at each step, thereby suppressing hallucinated continuations without discarding potentially useful information from high-uncertainty tokens. Without retraining, auxiliary models, or stochastic multi-view sampling, DVAD uses a single uncertainty-guided anchor view and can be implemented as a batched dual-stream decode with only marginal wall-clock overhead. Experiments on CHAIR, THRONE, and AMBER across multiple MLLM backbones demonstrate that DVAD consistently reduces object hallucination while preserving caption coverage and quality. Additional evaluations on MMBench and MMStar further show that these gains do not come at the cost of general vision-language understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NrfQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dual-View Anchored Decoding (DVAD), a training-free method for mitigating object hallucinations in multimodal large language models. It constructs an uncertainty-suppressed anchor view by masking visual tokens with high KL divergence from the image-level consensus, while preserving the original full view. The two views are decoded jointly, and their logits are fused using an architecture-dependent coefficient: positive for dense-token models and negative for sparse-token models. Experiments on three MLLMs demonstrate reductions in object hallucination with relatively low inference overhead.

Critical strengths
  • A simple and efficient training-free decoding framework. DVAD combines uncertainty-guided visual token masking with dual-view decoding without requiring additional training. Experiments across three MLLMs show consistent hallucination reductions with limited reported inference overhead.

  • Evaluation of the hallucination–coverage trade-off. The paper evaluates both hallucination reduction and object coverage across multiple benchmarks, rather than focusing solely on hallucination rates. The ablation studies further examine the effects of different masking strategies and decoding components, providing useful evidence for the effectiveness of the proposed design.

Critical weaknesses
  • Limited novelty compared with existing decoding methods. The observation that high-uncertainty visual tokens may contain useful information has already been discussed in Dropout Decoding. Moreover, DVAD's logit fusion largely follows the established VCD formulation, particularly when the fusion coefficient is negative. Thus, the method appears to be an incremental combination of existing techniques.

  • Insufficient validation of the architecture-dependent fusion strategy. The main distinguishing design of DVAD is the use of opposite fusion directions for dense- and sparse-token architectures. However, experiments on only one dense-token and two sparse-token models provide limited evidence that token compression explains this distinction. Additionally, the reported efficiency advantage over Dropout Decoding may partly result from comparing batched decoding with sequential execution.

Questions for authors
  • Novelty Beyond Existing Decoding Methods. Given the overlap with Dropout Decoding and VCD, could the authors clarify the distinctive methodological contribution of DVAD beyond combining uncertainty-guided masking with conventional dual-view logit fusion?

  • Validation of the Fusion Strategy and Efficiency. Could the authors provide more controlled experiments to validate the architecture-dependent sign selection, particularly across additional architectures or different visual-token compression levels? Additionally, how does DVAD's inference efficiency compare with Dropout Decoding under matched batching implementations?

1 comment

I haven’t seen a response yet.

Reviewer_HdDUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DVAD is a training-free decoding method that combines an original image view with an auxiliary view in which high-divergence visual tokens are zeroed. A probe prefill scores each visual token by the KL divergence between its vocabulary distribution and the mean distribution over visual tokens. The two views then decode with the same generated prefix, using a signed linear combination of their next-token logits. The method moves toward the masked view for LLaVA-1.5 and away from it for MiniGPT-4 and mPLUG-Owl2, motivated by the difference between dense spatial tokens and compressed visual interfaces. Experiments report reduced object hallucination with broadly preserved coverage, alongside a batched implementation intended to keep latency low.

Critical strengths
  1. The intervention is simple and well specified. It needs no training or additional model, and the appendices explain the masking location, extra probe pass, decoding loop, and fixed settings clearly. Keeping the original view available is a sensible way to avoid treating every unusual visual token as disposable (Sections 3.1–3.3, pp. 4–5; Appendices B–C, pp. 12–14).

  2. The empirical results show a useful, if moderate, improvement across the three tested backbones. On LLaVA-1.5, sentence-level CHAIR falls from 43.9 to 40.1 and instance-level CHAIR from 13.1 to 11.5, while AMBER coverage remains at least as high as greedy decoding. The random- and entropy-mask comparisons also help assess the choice of uncertainty score rather than only comparing against an unmodified model. Reporting matched samples and variation across three subsets is helpful (Tables 1 and 3, pp. 6 and 8).

Critical weaknesses
  1. The key dense-versus-sparse explanation is not tested as directly as the paper's framing suggests. The fusion rule is closely related to existing two-view contrastive decoding, and the uncertainty score is explicitly inherited from prior work; the most distinctive claim is therefore that the visual interface determines which sign to use. Yet the sign/ratio sensitivity study, anchor-only comparison, and mask-selection ablations are all on LLaVA-1.5. There is no reported comparison of both signs or anchor-only decoding on the two compressed backbones, nor a direct test showing that the selected tokens carry misleading evidence in one regime and useful evidence in the other. Likewise, growing textual context is offered as the grounding-attenuation explanation without a corresponding length- or position-dependent analysis. The results establish a working configuration on these checkpoints, but leave the mechanism and the sign rule for a new backbone uncertain (Section 3.3, pp. 5–6; Figures 3–5; Appendix H, p. 18).

  2. Some comparisons do not isolate the claimed practical advantages. The 1.02× latency figure includes the probe, which is good, but DVAD is batched while Dropout's three streams are executed sequentially; the 4.17× versus 1.02× comparison therefore includes an implementation advantage that could also be offered to the baseline. Without generated-token counts, matched-length timing, or memory/throughput results, it is also difficult to judge how far the near-zero-overhead claim extends. Separately, the general-capability comparison restricts DVAD's answer to option-letter token IDs while parsing unconstrained baseline generations. Those are different decision rules, so the small accuracy improvements do not cleanly demonstrate preserved capability under the decoding intervention (Table 4, p. 8; Appendices D.4, E.2, and G, pp. 16–18).

Questions for authors
  1. Could you report positive- and negative-coefficient sweeps, plus anchor-only and random-mask controls, on MiniGPT-4 and mPLUG-Owl2? Evidence connecting the selected tokens to supported and unsupported objects would make the proposed sign rule much more convincing. How would the sign be selected for an unfamiliar interface before inspecting evaluation performance?

  2. How much of the latency advantage remains when Dropout's streams are also batched? Please include output lengths, fixed-length tokens/second, timing variation, and peak memory, as well as at least one higher-throughput or shorter-output setting. This would distinguish low single-request latency on the reported A100 setup from a generally low computational cost.

  3. Please rerun MMBench/MMStar with the same option-letter restriction for every method. Also, can you provide official THRONE extraction results, or describe the current numbers consistently as CHAIR-extracted class-wise metrics? Appendix D.2 clearly discloses the substitution, but the main text presents THRONE as a separate benchmark even though it reuses the same COCO captions, object vocabulary, and extracted sets.

Reviewer_hG9LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dual-View Anchored Decoding (DVAD), a training-free method for reducing object hallucination in VLMs. It constructs an anchor view by masking high-uncertainty visual tokens, then combines the logits from the original and anchor views during decoding. The method uses positive fusion coefficients for dense-token models and negative coefficients for sparse-token models. Experiments on LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2 show reductions in CHAIR_S from 43.9 to 40.1, 36.6 to 33.4, and 54.2 to 51.1, respectively. The paper also reports a small latency increase from 2.44 to 2.48 seconds per image on LLaVA-1.5.

Critical strengths
  1. The uncertainty-guided anchor view provides a simple and intuitive way to mitigate object hallucination without additional training or auxiliary models.
  2. The experiments demonstrate consistent improvements in object hallucination metrics across three vlm backbones, while largely preserving caption recall and coverage.
  3. The batched dual-stream implementation offers a practical approach to limiting additional wall-clock latency.
Critical weaknesses

Limited evidence for the necessity of dual-view fusion. The component ablation in Figure 4 shows that anchor-only decoding already reduces CHAIR_S from 43.9 to 40.9, whereas full DVAD achieves 40.1. This suggests that most of the improvement comes from the anchor construction rather than the proposed fusion mechanism.

Insufficient justification for the sign-dependent fusion strategy. A central design choice is to use positive coefficients for dense-token models and negative coefficients for sparse-token models. However, the paper does not provide systematic comparisons of both fusion directions across the evaluated architectures. It remains unclear whether this distinction reflects a generalizable property of visual-token representations or an empirical choice specific to the tested backbones. A more comprehensive coefficient ablation and evaluation on unseen architectures would strengthen this claim.

Limited generalization evidence on modern VLMs. The evaluation is restricted to LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2, which are substantially older than current multimodal models. It is unclear whether the method remains beneficial when applied to stronger models with more advanced visual grounding capabilities. Evaluating recent models such as Qwen3.5, Qwen3.8 or Nemotron Omni would improve the practical relevance of the paper and provide a stronger test of the proposed dense/sparse fusion hypothesis.

Questions for authors

Please address my questions on the weaknesses. Also, how is the sign of the fusion coefficient determined for a previously unseen model? Can the authors demonstrate that the proposed dense/sparse criterion reliably predicts the preferred fusion direction without per-model tuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.