acceptodds
Withdrawn from ICLR 2027

Commitment Is Not Grounding: Visual-Semantic Rereading for Diffusion MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion multimodal large language models iteratively resolve answer tokens, yet committing to an answer does not establish visual support for the queried target. We introduce Internal Visual-Semantic Rereading (IVSR), an inference-time controller that audits the native answer at its first valid transfer. A maturity gate selects low-margin commitments for a short visual reread. The reread answer state retrieves an internal visual region, while a contextual question-target representation evaluates its support. A calibrated policy combines this alignment with reread answer margins to keep, reject, or recover the native answer. Backbone weights and prompts remain unchanged; thresholds use labeled held-out calibration. Across DiffusionVL, MMaDA, and LaViDa, IVSR improves all 18 POPE settings by 2.34 F1 points on average and reduces AMBER-D existence hallucination rates by 16.1–47.7%, with 1.4–9.8% measured gated latency overhead. Mechanism diagnostics and ablations support separating inspection priority, answer preference, and target-specific visual evidence in finite-answer verification.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion multimodal language models, hallucination mitigation, inference-time auditing, visual grounding, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Pd3eOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes IVSR, an inference-time method that selectively revises answers in diffusion MLLMs at the first valid Yes/No answer transfer without updating the backbone weights. IVSR selects low-margin answers for visual rereading. It then jointly considers (1) the Yes/No margins after rereading and (2) the cosine similarity between a visual region retrieved by the reread answer state and the contextual target representation in the question. The native answer is changed only when both evidence conditions are satisfied.

Critical strengths

Without updating the backbone weights, IVSR improves over native decoding on all 18 POPE settings across three diffusion MLLMs (Table 1). Its selective inspection keeps the measured additional latency to 1.4–9.8% (Table 2). The authors openly report harmful edits and acknowledge limitations in the available reproduction details.

Critical weaknesses
  1. The main comparison uses unequal calibration conditions. IVSR uses 1,200 labeled examples for calibration and configuration selection, while the baselines use their reported defaults. The matched-calibration comparison covers only one setting, where the advantage over the strongest tuned baseline is 0.40 F1 points.

  2. A simple baseline is missing. In the diagnostic setting reported in Table 9, IVSR increases the Yes ratio and recall while lowering precision. Simply adjusting the Yes/No decision threshold on the same calibration data might produce a similar effect, but this comparison is not provided.

  3. The region–target cosine score is insufficiently validated as a measure of target identity. The target-evidence ablation supports the usefulness of this component, but does not directly show whether e distinguishes similar objects such as a fork and a knife. Compatible hidden spaces alone do not establish this interpretation.

  4. The diagnostic evidence has limited scope. The Section 3 diagnostics use only one model and one setting. The spoon example (Figure 2, right) was selected after observing a beneficial change, and its margin magnitude of 0.125 is only one BF16 representable increment at the reported logit scale; full-precision behavior remains untested (Appendix D.4.2). In the persistence experiment, 266/267 initially correct preferences also persist. This supports the claim that errors can persist, but does not establish an error-specific mechanism.

  5. Reproducibility is insufficient. The benchmark controller’s tensor-level reread operation is not fully specified, some fitted threshold values are missing, and the profiling hardware is unspecified. The separately implemented diagnostic probe does not resolve the missing details of the benchmark controller.

  6. The paper is difficult to follow because of excessive terminology. Existing concepts receive new names, such as “commitment maturity” for the answer margin, and overlapping terminology makes the method harder to understand.

Questions for authors
  1. Can the authors provide matched-calibration comparisons with the baselines for the other backbones and settings?
  2. How does IVSR compare with simply adjusting the Yes/No decision threshold on the same calibration data?
  3. Does e actually distinguish similar objects? For example, what are the e values for the fork/knife and suitcase/backpack pairs in Figure 3, and how well does e alone separate correct from incorrect answers?
  4. Does the spoon result (Figure 2, right) persist in full precision, and can the authors provide full-precision results on more examples?
Reviewer_jZ9xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual hallucinations in diffusion MLLMs and finds that early decoding errors often persist unless the model revisits visual evidence. The authors also show that attention patterns alone cannot reliably distinguish correct from incorrect predictions. To address this, they propose Internal Visual-Semantic Rereading (IVSR), a training-free method that triggers additional visual rereading when the initial prediction has a low confidence margin. IVSR uses both answer confidence and region-level visual evidence to decide whether to keep, reject, or correct the initial answer. Experiments on DiffusionVL, MMaDA, and LaViDa across POPE, MME, and AMBER-D show consistent gains in accuracy and F1 over existing inference-time methods, with only 1.4%–9.8% additional latency.

Critical strengths
  1. Well-motivated design for diffusion decoding : Identifying the first valid answer transfer step t⋆t^\star as the intervention point is a reasonable choice for masked diffusion MLLMs. The analyses in Figures 1–3 provide useful evidence that early prediction errors tend to persist and that confidence gating and semantic verification should be handled separately. 
  2. Effective visual-semantic verification: The observation that correct and incorrect predictions often attend to similar image regions is interesting. Rather than relying on attention alone, IVSR uses the reread answer state to select relevant visual tokens and checks their semantic alignment with the question target. 
  3. Good accuracy-efficiency trade-off: IVSR improves performance across all 18 POPE settings and reduces hallucination rates on AMBER-D by 16.1%–47.7%. Since rereading is triggered for only around 9%–19% of samples, the additional latency remains low (1.4%–9.8%), making it considerably more efficient than baselines such as VCD and PAI.
Critical weaknesses
  1. Limited evaluation beyond binary QA: The current experiments mainly focus on Yes/No questions, which makes it difficult to judge how well IVSR generalizes to other tasks. It would be useful to include results on general VQA benchmarks or out-of-domain datasets, such as medical image VQA, to better understand the applicability of the method. 
  2. Missing implementation details: Some important details of the rereading operator Rθ\mathcal{R}_\theta and target selector Ω(q)\Omega(q) are missing, including the visual-access modifications, mask/cache updates, and token-matching rules. These details are necessary for reproducing the method. 
  3. Insufficient analysis of experimental stability: The paper does not report results across different random seeds or decoding temperatures. Since some improvements over the baselines are relatively small, it is unclear whether these gains are consistent or partly due to randomness. The authors should report the mean and standard deviation over multiple runs and examine how performance changes with different temperature settings.
Questions for authors

See the weaknesses. I have been struggling between weak rejection and weak acceptance for a long time, and I have decided that it is weak rejection. I hope the author can add corresponding experimental results based on my review comments during the rebuttal period and update them in different colors in the article for my subsequent review and score improvement.

Reviewer_y3KjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studied visual hallucination in masked-diffusion MLLMs from the angle of answer commitment. That is, the first valid transfer of the answer token exposes the candidate the native decoder is about to submit, and the authors argue this is the right moment to audit it. To this end, this paper proposed IVSR, a training-free inference-time controller, use a fixed K-step reread produces updated logits and hidden states and design a calibrated conjunctive policy keeps, rejects, or recovers the native answer. The authors conduct extensive experiments on three diffusion backbones (DiffusionVL, MMaDA, LaViDa) and the results show that the proposed method improves the F1 score while reducing the hallucination rate, at the cost of a manageable increase in processing time.

Critical strengths
  1. The method is training-free and shows consistent gains across all 18 POPE settings and three architecturally distinct diffusion MLLMs.
  2. The gated IVSR introduced only 1.4%–9.8% latency, far less than the approximately twice the latency introduced by the VCD/PAI method.
Critical weaknesses
  1. The authors could incorporate some newer hallucination mitigation methods (published as of 2026) into the comparison to better demonstrate the advantages of their proposed approach.
  2. In Figure 5, the proposed method performs significantly worse than the classical method VCD on some benchmarks. The authors could further analyze these failure modes.
  3. Regarding reread-length sensitivity, it is worth further analyzing why an excessive length is suboptimal. Theoretically, more extensive rereading should yield a richer recall of information.
Questions for authors

See Weaknesses

Reviewer_2mygOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes IVSR, an inference-time controller for diffusion MLLMs. It selects low-margin Yes/No answers for a short reread, then uses answer margins and region–target similarity to decide whether to change the answer. Results improve across three backbones, with relatively small reported latency overhead.

Critical strengths

Clear motivation. Committing to an answer does not mean the image supports it.

Consistent gains across all 18 POPE settings and lower AMBER-D existence hallucination rates.

Useful ablations and efficiency measurements. I also appreciate the reporting of harmful edits.

Critical weaknesses

The core reread operation is not fully specified. Appendix A.3 lists missing details about visual access, cache/mask updates, and readout synchronization. Some deployed thresholds are also unclear. These details are needed to reproduce the method.

The grounding claim needs stronger evidence. The target-score ablation shows that the score helps, but not necessarily that it measures visual support. The attention examples and the separate visual-access probe do not directly validate the full controller.

Simpler baselines are missing. In Table 9, recall improves but precision drops. There are 107 helpful edits and 90 harmful ones. How much of the gain could come from simply tuning the Yes/No decision threshold? A tuned margin-only controller would help answer this.

The matched-calibration comparison still has unequal search spaces: 16 configurations per baseline versus potentially 256 for IVSR. The gain over tuned PAI is only 0.40 F1 points, with no uncertainty estimates.

Evaluation is limited to binary answers. The diffusion-specific benefit of checking at first transfer is also not isolated.

Questions for authors

How are calibration sets constructed? Are they disjoint at the image level? Which settings transfer across benchmarks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.