acceptodds
Under review as a conference paper at ICLR 2027

OVERTURE: Decode Before the Echo for dMLLM

Abstract

Existing diffusion-based multimodal language models (dMLLMs) typically order decoding by prediction confidence. This strategy prioritizes high-confidence tokens, providing more reliable context for later predictions, but overlooks how decoding timing affects multimodal decision reliability. Analyzing complete denoising trajectories, we find that confidence-based decoding finalizes simple, linguistically predictable tokens first, while deferring more difficult tokens whose interpretation requires precise cross-modal alignment and richer semantic descriptions than those conveyed by formulaic, template-like language. As finalized text accumulates, linguistic context increasingly shapes later predictions, creating a growing linguistic echo, whereas visual evidence exerts less influence on candidate competition. For these visually dependent and more difficult tokens, reliable decisions require stronger visual evidence. Yet confidence-based decoding postpones their commitment until visual evidence exerts weaker influence, creating a difficulty–timing mismatch between evidence requirements and decision timing. This mismatch makes visually dependent decisions more vulnerable to erroneous final predictions. Tracking each token's attention context across denoising steps reveals a stable local Undershoot phenomenon. Pronounced Undershoot is concentrated among tokens associated with subsequent erroneous decisions, providing an observable signal for locating vulnerable generation decisions. We propose OVERTURE, a training-free, trajectory-guided priority decoding method that identifies vulnerable content from Undershoot and prioritizes the corresponding positions during regeneration while re-predicting tokens from the current generation state. Experiments across multiple benchmarks and model backbones demonstrate the broad effectiveness of OVERTURE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion multimodal large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AkGvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes OVERTURE, a training-free decoding method for diffusion-based multimodal language models. It detects trajectory deviations termed “Undershoot” during an initial generation pass, then regenerates the response while prioritizing selected positions and deferring EOS/EOT commitment. The paper motivates this approach through a proposed difficulty–timing mismatch and reports improvements over native decoding on two multimodal backbones.

Critical strengths
  1. The relationship between commitment order and visual grounding is an interesting research question.
  2. The method requires no additional training and shows improvements over native decoding, particularly on MMaDA.
Critical weaknesses
  1. Limited model coverage and insufficiently challenging evaluation. The multimodal evaluation is restricted to two early 8B backbones, MMaDA-8B-Base and LaViDa. More importantly, Appendix B reports that LLaDA-V does not exhibit the stable advancing frontier required by the detector and consequently excludes it from intervention evaluation. This raises a concrete concern about architectural generality. Although AMBER and DetailCaps provide useful additional coverage, the evaluation would benefit from tests that more directly distinguish visual grounding from textual shortcuts, such as MathVerse or MMMU-Pro.
  2. The baseline comparisons do not convincingly establish a competitive advantage. Section 4.1 explicitly states that external baselines are reported under their respective evaluation settings, whereas only Original and OVERTURE are evaluated under matched settings. Consequently, the rankings in Table 1 are difficult to interpret. Even taking the reported numbers at face value, OVERTURE trails the best compared method on five of the six shared metrics on LaViDa. For example, its MathVista and LLaVA-Bench-COCO scores are 41.80 and 70.60, compared with CACD’s 45.70 and 88.70. AMBER and DetailCaps lack comparisons beyond the native scheduler.
  3. Missing ablations prevent attributing the gains to Undershoot-guided prioritization. OVERTURE combines trajectory-based sentence selection, priority decoding, and EOS/EOT deferral. Section 3.1 states that EOS/EOT deferral remains active even when no sentence is selected. However, the paper does not report EOS-deferral-only or priority-only ablations, nor comparisons with random or confidence-based sentence selection. Matching the maximum generation length does not control for differences in actual response length or premature termination. The reported improvements therefore do not isolate the contribution of the proposed trajectory signal. Section 4.4 also claims weaker downstream performance with Top-2 selection without providing the corresponding quantitative results.
  4. The central mechanistic claims are insufficiently substantiated. Section 2.3 claims that Undershoot remains associated with errors after controlling for position, generation stage, and confidence, but the corresponding control procedure and quantitative results are not reported. Similarly, Section 2.4 defines a sensitivity measure and describes increasing textual influence, decreasing visual influence, and recoverable visual information, yet does not provide the corresponding quantitative trends, perturbation details, or probing protocol. These omissions make it difficult to assess whether the proposed difficulty–timing mismatch is supported by systematic evidence.
Questions for authors
  1. Can the method be evaluated on newer or architecturally more diverse dMLLMs and on benchmarks that more directly test dependence on visual evidence? How is applicability determined when a model lacks a stable advancing frontier, as observed for LLaDA-V?
  2. Can AdaBlock, Wavefront, and CACD be evaluated using the same examples, prompts, decoding settings, and evaluation pipeline as OVERTURE? Under this unified protocol, does OVERTURE offer a consistent advantage, particularly on LaViDa?
  3. Can you provide EOS-deferral-only and priority-only ablations, together with random and confidence-based sentence-selection controls? Please also report actual response lengths, truncation rates, and quantitative Top-1 versus Top-2 intervention results.
  4. Can you provide the quantitative text/vision sensitivity results, visual-information probing protocol, and the position-, stage-, and confidence-controlled analysis described in Sections 2.3–2.4? How do these results distinguish the proposed mechanism from alternative explanations?
Reviewer_SRqzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the difficulty–timing mismatch in diffusion-based multimodal language models (dMLLMs), arguing that confidence-based decoding may delay visually dependent tokens until accumulated linguistic context dominates their predictions. The authors introduce Undershoot, a trajectory-based signal derived from attention-context changes, and propose OVERTURE, a training-free method that prioritizes vulnerable token positions during regeneration. Experiments on MMaDA and LaViDa report improvements over their native decoding schedulers.

Critical strengths
  1. The paper presents an interesting perspective on the relationship between token commitment timing and multimodal hallucination. Investigating decoding trajectories to identify vulnerable generation decisions is potentially valuable. The training-free approach is also appealing, and some benchmark improvements are promising.
Critical weaknesses
  1. Missing fundamental decoding configurations and insufficient experimental validation.

The paper fails to report several essential dMLLM decoding hyperparameters, including the number of denoising steps, diffusion block size, denoising steps per block, token commitment schedule, and specific sampling configurations. The EOS/EOT handling and termination conditions of the original schedulers are also unclear.

These are not minor implementation details. Such parameters can substantially change generation quality and token commitment trajectories. Since the paper specifically investigates decoding timing, these configurations directly affect the phenomena being studied. Their omission makes the experimental results difficult to trust and raises serious concerns about whether the experimental design adequately accounts for the fundamental decoding mechanisms of dMLLMs.

Furthermore, the main paper contains only two experimental result tables: Table 1 reports benchmark performance, while Table 2 varies the maximum generation length. Even these results are difficult to interpret without the actual decoding configurations. For example, the authors do not specify the diffusion step count (T), although Figure 2 appears to illustrate a 64-step process. If (T=64) is used for generation lengths of 16, 32, and 64 in Table 2, how are the denoising steps and token commitment budgets allocated? These choices could substantially alter the decoding behavior.

Beyond these tables, the main paper provides little experimental evidence validating the proposed method itself. Although the appendix includes additional analyses and qualitative examples, convincing controlled experiments demonstrating the necessity and effectiveness of the proposed mechanism are still missing. For a paper introducing a new decoding strategy, I find the experimental validation seriously inadequate. The presentation also suffers from poor space allocation, with extensive methodological descriptions but very limited experimental analysis in the main paper.

  1. The contribution of Undershoot-guided decoding is not established, and the comparisons are inadequate.

OVERTURE introduces multiple modifications simultaneously, including a second generation pass and an additional EOS/EOT commitment-deferral rule. However, the paper provides no EOS/EOT-only baseline or comparisons against simple alternatives such as random or confidence-based prioritization. Therefore, it is unclear whether Undershoot itself contributes to the reported improvements.

The comparisons with AdaBlock, Wavefront, and CACD are also not conducted under a unified experimental protocol. Moreover, OVERTURE performs worse than existing methods on several important LaViDa metrics, making its advantages over established decoding strategies questionable.

The central mechanism is also insufficiently justified. The paper does not convincingly demonstrate that Undershoot identifies visually grounded errors better than general uncertainty signals. Furthermore, Stage 2 transfers only token positions from Stage 1, even though sentence boundaries and semantic content may change during regeneration. Thus, the connection between identifying vulnerable content and actually correcting errors remains unclear.

Overall, the experiments fail to establish whether Undershoot-guided prioritization is responsible for the reported improvements or offers meaningful advantages over simpler decoding modifications.

Questions for authors

1.What are the exact block sizes, denoising steps, steps per block, commitment schedules, and sampling configurations used in each experiment? How is (T) configured across the different generation lengths in Table 2, and how are EOS/EOT tokens handled in the original and proposed schedulers?

2.Can the authors provide controlled comparisons against EOS/EOT deferral alone, random prioritization, and confidence-based prioritization to isolate the contribution of Undershoot?

3.How do the authors ensure that vulnerable positions identified in Stage 1 correspond to the same semantic content during Stage 2 regeneration? Is there evidence that these positions are actually committed earlier and that this leads to error correction?

4.Can the authors provide comparisons with AdaBlock, Wavefront, and CACD under identical decoding and evaluation settings? What specific advantages does OVERTURE offer over these methods?

Reviewer_dKZPOpenReview ↗
Rating
2(−1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author proposed OVERTURE, a training-free two-pass decoding method for diffusion multimodal language models. It detects backward deviations of the dominant attention-context-change peak from a fitted trajectory, maps those steps to committed sentence positions, and prioritizes one sentence’s coordinates during full-MASK regeneration while deferring EOS/EOT. It reports gains over native schedulers on MMaDA and LaViDa across reasoning, hallucination, and captioning tasks, with qualitative diagnostics and a mixed text-only extension.

Critical strengths
  1. The paper connects trajectory analysis with a practical, training-free decoding intervention. Using generation history to guide commitment order is an interesting approach that could improve existing models without retraining.

  2. The reported improvements over native schedulers cover two multimodal backbones and several task types. This provides encouraging evidence of practical value beyond a single model or benchmark.

Critical weaknesses
  1. The experiments do not isolate the contribution of Undershoot-guided priority from EOS/EOT deferral and regeneration. Without component controls, it is unclear whether the proposed signal is responsible for the improvements, which limits support for the main algorithmic contribution.

2.The central diagnostic and mechanism claims lack sufficient quantitative support. The selected-sentence error rates have no random or confidence-based comparison, and the claimed shift from visual to linguistic influence is not backed by detailed results. This makes it difficult to judge whether Undershoot reliably identifies the proposed failure mode.

  1. The implementation code is not provided.
Questions for authors
  1. Contribution of the individual components

Could you compare OVERTURE with the original scheduler using exactly the same EOS/EOT deferral rule? Under the same regeneration and EOS settings, how does Undershoot selection compare with random or low-confidence sentence selection? Please also report actual output lengths and total inference cost.

  1. Evidence for the central diagnostic and mechanism claims

Could you provide the controlled error analysis described in Section 2.3, including sample counts, the control or matching procedure, effect sizes, and uncertainty estimates? For Section 2.4, could you show the quantitative text/image intervention results and explain how visual recoverability is measured? How do you account for the fact that positions still unresolved at later steps may be different from those resolved earlier?

  1. Evaluation protocol

Section 4.1 states that each benchmark uses a fixed 1,000-example subset. How does this apply to LLaVA-Bench-COCO, whose standard version contains 90 questions? Please clarify the dataset version, the number of unique images and questions, and whether examples are repeated or expanded. Also, which external baseline results were rerun under the same settings, and which were taken from other reports? Since the paper states that external baselines use their own evaluation settings, it is important to know which comparisons are directly comparable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.