acceptodds
Under review as a conference paper at ICLR 2027

VIVA: Visual Information Dispersion via Attention Guidance for Parallel Decoding in Diffusion MLLMs

Abstract

Diffusion multimodal large language models (dMLLMs) enable parallel decoding by predicting multiple masked tokens using bidirectional context, offering a promising alternative to autoregressive MLLMs. However, existing token selection strategies largely determine which tokens to decode based on prediction confidence at each step, without considering how visual conditioning influences the overall decoding trajectory. In this work, we analyze dMLLM decoding trajectories and identify two key issues: visually relevant tokens tend to be decoded in later stages, limiting their contribution as context for earlier predictions, and tokens describing visual content often influence each other's predictions, making their simultaneous decoding prone to prediction errors. Based on these observations, we propose **V**isual **I**nformation Dispersion **v**ia **A**ttention Guidance (VIVA), an inference-time decoding method that disperses visually informative tokens both across decoding steps and within each step. VIVA uses token-to-image attention to distribute visually relevant tokens across the decoding trajectory and limits their simultaneous selection within each step, while preserving the ability to decode tokens in parallel. Across six benchmarks and three open-source dMLLMs, VIVA demonstrates strong performance against existing decoding baselines, particularly when multiple tokens are decoded in parallel. Specifically, on LaViDa-Reason, VIVA improves CIDEr from 67.5 to 86.0 on COCO Captioning and MMBench accuracy from 43.0 to 51.4 over confidence-based decoding when decoding eight tokens per step. Further analyses and qualitative examples confirm that VIVA effectively mitigates both identified decoding issues.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion multimodal large language models, Parallel decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_phHfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VIVA, a training-free decoding strategy for diffusion multimodal large language models (dMLLMs). The method is motivated by two observations: (1) visually informative tokens tend to be decoded at later stages under confidence-based decoding, and (2) jointly decoding tokens with high image attention leads to larger parallel prediction errors. To address these issues, VIVA incorporates image attention into token selection to prioritize visually informative tokens and introduces hierarchical constraints to distribute visually informative tokens across decoding steps. The method also groups adjacent tokens according to their image-attention distributions to limit the simultaneous decoding of potentially dependent tokens.

Critical strengths

S1. The method demonstrates performance improvements over existing decoding strategies, particularly under highly parallel decoding settings.
S2. VIVA is training-free and introduces relatively small inference overhead, making it applicable to existing dMLLMs without model modifications.
S3. The analysis of decoding trajectories and image attention provides an interesting perspective on token selection in diffusion MLLMs.

Critical weaknesses

W1. Insufficient justification for prioritizing visually informative tokens.

The paper assumes that visually informative tokens should be decoded earlier so that they can provide useful context for subsequent predictions. However, the motivation for this assumption is not sufficiently established. Since visual information is already available throughout the decoding process, it is unclear why committing high-image-attention tokens earlier would necessarily improve prediction quality. Moreover, high image attention does not imply high prediction confidence, and prematurely committing uncertain tokens could potentially introduce errors into subsequent decoding steps.

While the paper demonstrates that visually informative tokens tend to be decoded later, this observation alone does not establish that late decoding is harmful.

W2. Unclear connection between parallel prediction error and actual generation quality.

The paper motivates its method by showing that jointly decoding tokens with high image attention results in larger parallel prediction errors. However, these errors characterize the discrepancy between joint and factorized predictive distributions, rather than actual prediction errors with respect to ground-truth tokens.

A larger distributional discrepancy does not necessarily imply incorrect token predictions or degraded generation quality. Therefore, the paper does not sufficiently establish why tokens with high parallel prediction errors should be decoded separately.

Furthermore, the proposed segmentation assumes that tokens attending to similar visual regions exhibit stronger predictive dependencies. However, the analysis primarily examines image-attention mass rather than attention-distribution similarity, leaving the justification for attention-based segmentation unclear.

W3. Inconsistent improvements across tasks and limited understanding of failure cases.

Although VIVA achieves substantial improvements in several settings, its effectiveness varies considerably across models, tasks, and decoding budgets. For example, the improvements on the understanding benchmarks in Table 1 are less consistent across different backbones, while Table 2 also includes settings in which VIVA underperforms confidence-based decoding or VRCD.

The paper does not sufficiently analyze why the method performs well in some settings but degrades performance in others. A more detailed investigation of these failure cases would help clarify when the proposed strategy is effective and whether its underlying assumptions generalize across different tasks.

W4. Limited justification for the hierarchical attention constraints.

Equation (10) distributes the remaining attention mass according to the number of remaining decoding steps. However, the paper does not sufficiently justify why this allocation is appropriate for minimizing parallel prediction errors.

In addition, although the theoretical analysis provides a mutual-information-based upper bound on dependency error, replacing mutual information with image-attention mass is a heuristic approximation rather than a theoretically guaranteed bound. The connection between the proposed attention-based constraints and actual dependency reduction therefore remains insufficiently validated.

Questions for authors

Q1. Is there qualitative or quantitative evidence that decoding visually informative tokens earlier improves subsequent predictions? In particular, can the authors isolate the benefit of early visual-token decoding from that of reducing simultaneous token selection?

Q2. Why does VIVA show inconsistent improvements across tasks and decoding settings? What are the main causes of the performance degradation observed in some settings in Tables 1 and 2?

Q3. Can the authors provide empirical evidence that image-attention similarity correlates with actual inter-token dependency? How does the proposed attention-based segmentation compare with simpler alternatives, such as fixed-size or position-based grouping?

Q4. Why should larger parallel prediction errors necessarily lead to lower generation accuracy? Can the authors directly analyze the relationship between the proposed dependency-error metric and actual token prediction errors?

Q5. What motivates the allocation rule in Equation (10)? Have the authors compared it against alternative allocation strategies or thresholds?

Reviewer_TqBgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VIVA, an inference-time decoding method for diffusion-based multimodal large language models (dMLLMs). The authors observe that visually informative tokens tend to be decoded later, while decoding multiple visually dependent tokens in parallel can introduce prediction errors. To address these issues, VIVA uses token-to-image attention to prioritize visually informative tokens and introduces segment-level and sequence-level constraints to prevent them from being concentrated within the same decoding step.

The method is evaluated on three dMLLMs—LaViDa-Reason, MMaDA, and LLaDA-V—across six multimodal benchmarks. The authors report improvements over confidence-based decoding, particularly under highly parallel decoding settings.

Critical strengths
  1. An interesting approach to multimodal diffusion decoding.

The paper considers visual information and token dependencies when deciding which positions to decode. Combining image-attention guidance with constraints on parallel token selection is a reasonable direction beyond conventional confidence-based scheduling.

  1. A training-free method with promising results in highly parallel settings.

VIVA requires no additional training or model modifications. It achieves notable improvements on captioning and multimodal understanding tasks, particularly with LaViDa-Reason under aggressive parallel decoding, while introducing relatively little inference overhead.

Critical weaknesses
  1. The novelty of the motivating observations is limited, and several relevant studies are not adequately discussed.

The paper presents the early commitment of easily predictable tokens and the delayed decoding of visually informative tokens as key observations. It also highlights dependency errors arising from parallel token prediction. However, closely related phenomena have already been investigated in diffusion language model research.

In particular, Thinking Diffusion (Kim et al., CVPR 2026) analyzes the visual dependence of tokens generated at different denoising steps and reports that visual grounding is weaker in early stages and becomes stronger later. This closely overlaps with VIVA's first observation. Generation Order and Parallel Decoding in Masked Diffusion Models: An Information-Theoretic Perspective (Zhang et al., 2026) examines easy-first decoding and dependency errors in parallel token prediction. Similarly, Train for the Worst, Plan for the Best (Kim et al., ICML 2025) studies the importance of token ordering in masked diffusion models.

I acknowledge that these works do not propose the same multimodal decoding algorithm as VIVA. The attention-guided dispersion strategy and hierarchical constraints are different from the methods used in these studies. However, much of the underlying motivation, including decoding-order bias, delayed visual grounding, and dependencies in parallel decoding, has already been explored.

These related studies are not adequately discussed, and the paper does not clearly explain which parts of its observations are genuinely new. I would have liked to see a more careful distinction between previously established findings and the specific contributions of this work.

  1. The connection between image-attention mass and inter-token dependency is not sufficiently justified.

A central assumption of VIVA is that image-attention mass can serve as a proxy for inter-token dependency. However, a token's reliance on visual information does not necessarily imply that it strongly depends on other tokens. For example, two tokens referring to different image regions may both have high image-attention mass while being relatively independent.

The paper reports a correlation between image-attention mass and parallel prediction error, but this does not establish that attention mass reliably measures inter-token dependency. Similarly, the mutual-information analysis provides an upper bound on dependency error, but it does not directly justify approximating mutual information with image-attention mass.

This distinction matters because the proposed selection strategy uses attention mass to restrict which tokens can be decoded together. Without stronger evidence that this signal captures the relevant dependencies, it is unclear whether the method is actually addressing the claimed source of parallel prediction errors.

The empirical results also vary across backbones. While VIVA performs well on LaViDa, it underperforms VRCD or confidence-based decoding in several LLaDA-V settings. The paper provides little analysis of why the proposed attention-based strategy works better for some models than others.

  1. The necessity and effectiveness of the hierarchical constraints are not fully established.

Another key contribution of VIVA is its hierarchical selection mechanism, which combines segment-level and sequence-level constraints to distribute visually informative tokens across decoding steps. However, the experiments do not convincingly show that this relatively elaborate design is consistently better than simpler attention-guided reordering.

In Table 3, hierarchical constraints provide substantial gains under highly parallel decoding (r = 0.125), but they are not consistently beneficial at other decoding ratios. For example, on MMBench, unconstrained attention-guided decoding achieves 60.00 at r = 0.25 and 65.40 at r = 0.5, whereas the hierarchical version achieves 59.80 and 64.20, respectively.

More importantly, the method constructs segments based on changes in image-attention distributions, but the paper does not demonstrate that these segments actually capture groups of mutually dependent tokens. There is also no comparison with simpler alternatives, such as fixed-length segmentation or position-based grouping.

Since hierarchical dispersion is one of the main features distinguishing VIVA from straightforward attention-guided token selection, I believe its design needs stronger experimental justification. The current results suggest that it can help under certain conditions, but they do not establish why this particular segmentation and constraint mechanism is necessary.

Questions for authors

1.How do the authors distinguish VIVA's motivating observations from prior findings in Thinking Diffusion (Kim et al., CVPR 2026), Generation Order and Parallel Decoding in Masked Diffusion Models (Zhang et al., 2026), and Train for the Worst, Plan for the Best (Kim et al., ICML 2025)? What specific new insights does VIVA provide beyond these studies?

2.What evidence supports using image-attention mass as a proxy for inter-token dependency? Could the proposed constraints unnecessarily separate tokens that attend strongly to different image regions but have little dependency on each other?

3.Why do hierarchical constraints sometimes perform worse than unconstrained attention-guided decoding in Table 3? How does the proposed attention-based segmentation compare with simpler alternatives such as fixed-length or position-based grouping?

4.Why does VIVA underperform VRCD or confidence-based decoding in several LLaDA-V settings? Are these differences related to model architecture or the characteristics of image-attention distributions?

Overall, this work appears to address limitations of diffusion (M)LLMs that have already been identified in several prior studies, while relying on parallel decoding, a broadly established approach used across a wide range of domains and model architectures. Although VIVA introduces specific modifications to the token selection process, the paper does not convincingly demonstrate a fundamentally new problem, a sufficiently distinctive methodological contribution, or clear advantages over existing approaches. I would encourage the authors to clarify what genuinely novel insights or technical contributions this work offers beyond prior research, and why these contributions are sufficiently significant for acceptance at ICLR.

Reviewer_uRExOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which masked positions to commit during parallel decoding in diffusion multimodal language models. Confidence-based selection tends to leave high-image-attention tokens until late in decoding, and the authors find larger parallel prediction discrepancies when several such tokens are selected together. VIVA raises their priority using an attention-weighted confidence score, groups nearby positions by changes in image-attention distributions, and limits joint selection through segment- and sequence-level attention budgets.

The method requires no training. Experiments use three models, six benchmarks, and three ratios of decoding steps to generation length. The strongest gains occur under high parallelism, particularly for LaViDa-Reason captioning. The paper also reports component ablations, sensitivity analyses, a mutual-information interpretation, and end-to-end inference measurements.

Critical strengths
  1. Two clean, well-evidenced observations that open a new angle on dMLLM decoding. Prior decoding work selects tokens by certainty; this paper demonstrates, with controlled re-masking experiments on two model families, that visual informativeness is systematically back-loaded in the trajectory and that attention mass is a better predictor of parallel dependency error than confidence is. The observation that top-A selection maximizes dependency error while VIVA keeps it near the confidence baseline (Fig. 5) directly validates the dispersion design. This is a useful, transferable piece of understanding for the growing dMLLM literature.

  2. Strong empirical payoff precisely where parallel decoding hurts most, with disciplined ablations. The gains at r=0.125 are large and consistent across three models (24/27 settings improved vs. confidence), the Table 3 ablation cleanly separates attention-guided reordering from hierarchical constraints (showing the segment level is the key ingredient), the λ-sensitivity table shows robustness across an order of magnitude, and the cost analysis shows essentially unchanged memory and ~2–5% time overhead. The qualitative trajectories (Fig. 6) make the mechanism tangible.

Critical weaknesses
  1. The baseline set is too simple. Comparisons are limited to Confidence, Entropy, and VRCD (default configuration). Several trajectory-level methods the paper itself cites — PC-Sampler, LocalLeap, lookahead unmasking, EB-Sampler, information-gain sampling — operate on the same masked text stream and are directly pluggable into the dMLLM decoding loop, yet none is evaluated. Without such comparisons (or a combination study, since the signals are partly orthogonal), it is hard to tell how much of the gain exceeds what existing trajectory-aware selection already achieves.

  2. The benchmarks are basic perception suites and miss the long-output scenarios the method is designed for. VQAv2, DocVQA, and MMBench are short-answer tasks where most of the 128-token budget is padding — exactly the regime where the paper's two motivating phenomena are weakest. No long CoT reasoning (MathVista, MathVerse, WeMath, OlympiadBench), no comprehensive reasoning suites (MMMU/MMMU-Pro, MMStar), no long-document understanding (MMLongBench-Doc), and no open-ended faithfulness check (MMHal-Bench) is tested — despite both backbones (LaViDa-Reason, MMaDA-MixCoT) being trained for thinking-style outputs. The demonstrated wins are confined to short-to-medium captions (L = 16–32), leaving the central use case unverified.

  3. Gains are strongly bound to task and configuration, with unexplained regressions and no usage guidance. LaViDa DetailCaps falls below Confidence at r = 0.25/0.5; LLaDA-V Flickr30K falls at r = 0.125/0.5; MMBench at r = 0.5 with lambda = 20 is below Confidence. The paper neither explains these regressions nor states when VIVA should be preferred. The selection procedure for lambda (20 vs. 100 by task type) is unstated, the sensitivity analysis reuses samples from the evaluated distributions, and main tables lack confidence intervals.

  4. The theoretical justification rests on unvalidated links and small-sample observations. The substitution of image-attention mass for inter-token mutual information is only hypothesized: the sole support is a KL-divergence trend on 100 images (Table 6), which shows sensitivity to image removal, not dependency with other tokens. Only last-layer, head-averaged attention is used, with no layer sensitivity analysis; the tighter hierarchical bound in Appendix C.2 requires small cross-segment dependency, a condition never checked; and the remasked-sequence estimator of parallel prediction error is underspecified, leaving open the confound that high-attention tokens are simply lower-confidence content words.

Questions for authors
  1. Why were the cited trajectory-level decoding methods (PC-Sampler, LocalLeap, lookahead unmasking, EB-Sampler, information-gain sampling) not evaluated? Could you report at least PC-Sampler and LocalLeap under the same models, benchmarks, and r values — ideally also combined with VIVA?

  2. How does VIVA perform on long-output scenarios: CoT reasoning (MathVista, MathVerse, WeMath), comprehensive suites (MMMU/MMU-Pro), long-document understanding (MMLongBench-Doc), and open-ended faithfulness (MMHal-Bench)? Do the two motivating observations still hold on these longer generations?

  3. What explains the regressions on LaViDa DetailCaps (r = 0.25, 0.5), LLaDA-V Flickr30K (r = 0.125, 0.5), and MMBench at lambda = 20, r = 0.5? Under what conditions should VIVA be preferred over Confidence or VRCD? How were lambda = 20/100 chosen, and were the sensitivity-analysis samples disjoint from evaluation? Please add confidence intervals to the main tables.

  4. Beyond the 100-image KL trend (Table 6), what justifies attention as a proxy for inter-token dependency? How sensitive are results to the attention layer choice, and can you control the confound that high-attention tokens are typically lower-confidence content words (e.g., confidence-matched comparison of parallel prediction error)?

  5. How does JSD segmentation compare with equal-length or shuffled boundaries, and does VIVA still improve over a confidence baseline replaying its per-step token-count schedule under the same T? Also, please specify the dependency-error estimator used in Figures 2/4/5 (token order, conditioning, protocol per panel) and the meaning of k in Table 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.