acceptodds
Under review as a conference paper at ICLR 2027

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

Abstract

Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation. Starting from a fully masked sequence, dMLLMs progressively decode the sequence by unmasking a subset of the remaining masked positions at each step. Since the selected tokens serve as the prediction context for subsequent steps, deciding which tokens to decode is crucial to the quality of the final output. The most common strategy prioritizes tokens based on a certainty measure that tends to favor tokens frequently observed in the training data. Recent approaches instead order tokens according to their influence on subsequent predictions, but do not explicitly account for the input image. We propose the Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens. We further impose a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded subset. Extensive experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler, which outperforms the Info-Gain Sampler by an average of 19.3 CIDEr points across the captioning benchmarks and surpasses it on COCO Caption while using only half as many decoding steps.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion multimodal large language models, parallel decoding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zcUYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the Visual Information-Guided Sampler (VIG-Sampler), a training-free parallel decoding strategy for diffusion Multimodal Large Language Models (dMLLMs). To address the issue of conventional samplers prematurely predicting semantically uninformative tokens (such as punctuation or stop words) based on text-driven certainty, the authors propose leveraging token-to-image attention weights readily available during the decoding process. VIG-Sampler modifies token selection in two ways: first, it reweights token confidence based on image-attention mass to promote visual grounding; second, it introduces a similarity penalty to discourage the concurrent selection of tokens with highly overlapping image-attention distributions, thereby maximizing information gain.

Critical strengths
  1. The proposed method elegantly circumvents the severe computational bottleneck of the strong Info-Gain baseline, which requires multiple expensive forward passes to evaluate token combinations. By "free-riding" on the naturally computed self-attention matrices during a single forward pass, VIG-Sampler achieves superior parallel decoding performance with an inference time almost identical to the simplest confidence-based sampling.

  2. The preliminary analyses effectively bridge the gap between empirical phenomena and the proposed solution. The authors convincingly demonstrate that a token's image-attention mass correlates with its reliance on visual evidence, and that high attention similarity correlates with redundant information.

  3. The authors test the method across a diverse set of multimodal tasks and different model architectures. The consistent performance gains, particularly the finding that VIG-Sampler with k=8k=8 can match or surpass conventional samplers at k=2k=2 with fewer decoding steps, highlight its effectiveness.

Critical weaknesses
  1. While highly effective in practice, the core algorithmic contribution is essentially a combination of existing heuristics. Utilizing attention maps as a proxy for feature importance or visual grounding is a standard interpretability practice. Similarly, penalizing pairwise similarity to reduce redundancy is a well-known diversity-promoting technique (akin to Maximal Marginal Relevance). The novelty lies strictly in its application to dMLLM decoding schedules rather than representing a fundamental methodological innovation.

  2. The paper asserts that penalizing attention similarity increases sequence-level "information gain," which has a strict entropy-based definition in Equation 4. However, the connection between the cosine similarity of attention vectors and rigorous sequence entropy reduction is established solely through an empirical observation (Figure 2). The paper lacks a theoretical derivation or proof to explain why lower attention similarity bounds or mathematically approximates the reduction in predictive entropy.

Questions for authors

see weakness

1 comment

Weakness 1: Novelty beyond existing heuristics

Response: We appreciate the reviewer's perspective and acknowledge that prior research has explored the use of attention maps and similarity-based penalties. We would like to clarify that VIG-Sampler's novelty lies not in the use of these concepts per se, but in the new empirical insights and their translation into a decoding strategy that addresses the limitations of existing dMLLM samplers.

Specifically, to the best of our knowledge, we are the first to investigate the relationship between image attention and the informativeness of decoded tokens for subsequent predictions in dMLLM decoding. Our analysis reveals that (i) tokens with higher image-attention mass are more likely to be visually grounded and yield greater information gain, and (ii) tokens with similar image-attention patterns tend to exhibit greater information redundancy when decoded jointly. These findings naturally motivate us to adopt image-attention-based score reweighting and image-attention similarity regularization as two complementary design choices aimed at increasing set-level information gain.

We believe that our findings and their instantiation in VIG-Sampler, supported by extensive experimental validation, constitute a meaningful contribution to research on visually grounded parallel decoding for dMLLMs.

Weakness 2: Theoretical support for the connection between attention similarity and sequence-level information gain

Response: We appreciate the reviewer’s valuable comment that the connection between attention similarity and sequence-level information gain could be better substantiated. To address this concern, we complement our empirical findings with an information-theoretic explanation. We first discuss the potential information redundancy among tokens with similar image-attention patterns, and then explain its effect on sequence-level information gain.

Prior studies on attention in Transformer-based models have observed that tokens with similar attention patterns tend to have similar representations, i.e., similar hidden states [1, 2]. These findings support the assumption that two tokens with similar image attention may convey overlapping visual information and provide redundant context for subsequent predictions. To connect this potential redundancy to reductions in sequence-level information gain, we first examine its effect at an individual masked position. When the two masked positions in S=i,j\mathcal S=\\{i,j\\} are decoded in parallel, the expected token-level information gain at a remaining masked position k∈Mt(S)k\in\mathcal M_t^{(\mathcal S)} is expressed as the mutual information

I(yk;yi,yj)=H(yk)−H(yk∣yi,yj).I(y^k;y^i,y^j)=H(y^k)-H(y^k\mid y^i,y^j).

By the chain rule [3], this gain can be decomposed into the information provided by yiy^i and the additional information provided by yjy^j once yiy^i is known:

\begin{aligned} I(y^k;y^i,y^j) &=I(y^k;y^i) +I(y^k;y^j\mid y^i). \end{aligned}

If the two decoded tokens provide redundant information for predicting yky^k, the information provided by yjy^j after yiy^i is known can be smaller than that provided by yjy^j alone:

I(yk;yj∣yi)<I(yk;yj)I(y^k;y^j\mid y^i)<I(y^k;y^j).

Combining this condition with the chain rule gives

I(yk;yi,yj)<I(yk;yi)+I(yk;yj).I(y^k;y^i,y^j)<I(y^k;y^i)+I(y^k;y^j).

Thus, informational redundancy between the two tokens leads to their joint information gain being smaller than the sum of their individual gains.

As defined in our manuscript, SIR quantifies the effect of such redundancy on sequence-level information gain, with gains measured as reductions in average marginal predictive entropy. Using this measure, we found that token sets with more similar image-attention patterns had higher SIR values (Figure 2, right), indicating smaller joint information gain relative to the sum of individual gains. Together with the information-theoretic explanation above, this empirical finding provides a rationale for using image-attention similarity as a regularization term in parallel decoding.

We have incorporated this theoretical explanation into Appendix B.2 of the revised manuscript.


[1] Namuk Park et al. “What Do Self-Supervised Vision Transformers Learn?” ICLR, 2023.

[2] Zichen Liu et al. “Token Coordinated Prompt Attention is Needed for Visual Prompting”. ICML, 2025.

[3] Thomas M. Cover and Joy A. Thomas. “Elements of Information Theory”. Wiley, 1991.

Reviewer_RKXJOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VIG-Sampler, a training-free parallel decoding strategy for diffusion multimodal large language models (dMLLMs). At each unmasking step, instead of selecting tokens purely based on prediction confidence or text-only information gain, VIG-Sampler uses the last-layer token-to-image attention weights to (1) reweight token confidence by total image-attention mass so visually grounded tokens are prioritized, and (2) greedily penalize pairwise cosine similarity between the image-attention distributions of selected tokens to reduce information overlap during parallel decoding. Experiments on LaViDa, MMaDA, and LLaDA-V across four image captioning datasets and three VQA datasets show that VIG-Sampler maintains strong performance when decoding 4 to 8 tokens per step with minimal runtime overhead.

Critical strengths
  1. Training-free and lightweight sampler that maintains accuracy under large parallel decoding steps. Reweighting token confidence with image-attention mass and penalizing attention overlap requires no extra model forward passes and adds negligible latency over standard confidence sampling. While its accuracy at 2 tokens per step is comparable to existing samplers, its main advantage appears when scaling to 4 or 8 tokens per step, where it avoids the severe accuracy drop suffered by prior baselines and enables much faster decoding.
  2. Consistent improvements across multiple dMLLM backbones on captioning. The method works out of the box with fixed hyperparameters across LaViDa, MMaDA, and LLaDA-V, and the empirical connection between image-attention overlap and shared information ratio is clearly demonstrated.
Critical weaknesses
  1. Evaluation is limited to image captioning and simple VQA, lacking spatial and multimodal reasoning benchmarks. The current experiments only cover captioning and OCR/chart VQA. In these tasks, prioritizing decoded tokens that attend to specific image regions to extract visual entities is naturally effective. However, the paper does not evaluate on general multimodal benchmarks involving complex spatial relations or multi-step reasoning—such as MME, MMBench, or GQA—where the model also needs to attend to prompt and reasoning tokens rather than just local image patches.
  2. Penalizing image-attention similarity may hinder fine-grained attribute or relational reasoning. As shown in Figure 3, tokens such as "dog" and "brown" have high image-attention similarity because they refer to the same object, and are therefore discouraged from being decoded together. While forcing parallel tokens to attend to distinct image regions works well for enumerating different objects in captioning, it could hurt tasks that require describing multiple attributes of a single object or reasoning about fine-grained spatial relationships within the same region.
Questions for authors
  1. Performance on spatial relation and general reasoning benchmarks: Since the current evaluation focuses on captioning and OCR-oriented VQA where extracting visual entities dominates, how does VIG-Sampler perform on benchmarks that test spatial relations and broader multimodal reasoning, such as GQA, MME, or MMBench?
  2. Effect of the similarity penalty on multi-attribute descriptions: In Figure 3, "dog" and "brown" are penalized from being selected in the same step because both attend to the dog. When a task specifically requires generating fine-grained attributes or relational phrases about a single object, does the similarity penalty (lambda = 3) slow down or degrade the decoding of co-referring attribute tokens?
3 comments

Weakness 1 & Question 1: Evaluation on spatial relations and multimodal reasoning benchmarks

Response: We thank the reviewer for highlighting the need for evaluation on spatial relations and broader multimodal reasoning tasks. To address this concern, we extend our evaluation of VIG-Sampler to three additional benchmarks: MME, MMBench, and GQA. We conduct these experiments using MMaDA, a dMLLM with multimodal reasoning capabilities, with a generation length of 128 and a block length of 16 across all three benchmarks.

As shown in the table below, VIG-Sampler achieves consistent improvements across these benchmarks, further demonstrating its effectiveness on diverse multimodal understanding and reasoning tasks. We have incorporated these results into Appendix C.2 of the revised manuscript.

kkSamplerMMBenchMMEGQA
2Confidence51.51069.442.4
Info-Gain49.21046.043.0
VIG-Sampler54.61229.449.0
4Confidence37.1509.825.6
Info-Gain34.1600.528.6
VIG-Sampler49.2838.737.2
8Confidence9.973.215.8
Info-Gain9.988.514.8
VIG-Sampler21.2155.221.4

Weakness 2 & Question 2: Effect of similarity regularization on attribute and relational predictions

Response: We thank the reviewer for raising this concern. We acknowledge that tokens attending to the same image region may convey complementary information, such as an object and its attributes, and that similarity regularization may delay$^*$ the decoding of such tokens.

However, deferring such tokens is not necessarily a drawback, as it can mitigate prediction inconsistencies arising from parallel decoding. Specifically, in dMLLMs, each masked token is predicted without conditioning on other tokens predicted within the same decoding step. Consequently, simultaneously decoding an object token and its associated attribute tokens may lead to repetition or incorrect word order (e.g., "dog dog" or "dog brown" instead of "brown dog"). In contrast, decoding these tokens in separate steps allows the token decoded first to serve as context for predicting the other token, reducing the risk of such errors.

To assess the impact of similarity regularization on fine-grained attribute and relational predictions, we conduct an additional ablation study on DetailCaps by comparing λ=0\lambda = 0 with the original setting (λ=3\lambda = 3). DetailCaps evaluates detailed, long-form image captions describing multiple objects, their attributes, and inter-object relations. In addition to the overall CAPTURE score reported in the paper, we report Object, Attribute, and Relation F1 scores, which measure the correspondence between these elements in generated and reference captions.

The consistent improvements across all three F1 scores demonstrate that similarity regularization improves captioning performance without compromising attribute and relational predictions. We have incorporated these results into Appendix C.4 of the revised manuscript.

kkλ\lambdaObject F1Attribute F1Relation F1
2061.954.253.5
362.254.753.9
4054.444.949.0
359.853.253.2
8034.824.528.3
346.738.042.5

∗^* Since the number of tokens decoded per step is fixed in our experimental setup, delaying certain tokens does not increase the total number of decoding steps.

Reviewer_RKXJOpenReview ↗

Thanks for detailed responses

Thanks the authors for responses to each of my concerns. I suppose additional experiments fully resolve my questions. I raised my score.

We sincerely thank the reviewer for their positive feedback and thoughtful consideration of our responses. We also appreciate their constructive comments and suggestions, which helped us strengthen the manuscript.

Reviewer_RbP6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free token-selection approach, VIG-Sampler, for diffusion multimodal language models. It employs attention to reweight token confidence and penalizes generated tokens with similar attention patterns in the decoding step. The extensive experiments show the effectiveness of the proposed method.

Critical strengths
  1. The paper is well-organized and easy to follow.
  2. The proposed method requires no additional training and extra inferences, making it practical and lightweight to improve performance.
  3. The evaluation is comprehensive spaning from multiple models, decoding budget, generation lengths, and hyperparameters.
Critical weaknesses
  1. Higher attention score does not mean the prediction word is accurate because a model would misidentify objects or attributes. Are there any quantitative results to verify this?
  2. Although the predicted words look at the same image region, it does not mean the words provide the same-level information. For example, "red" and "car" may attend to the same object while describing different properties; penalizing their attention similarity could delay useful words. It would be helpful to comprehensively test whether the penalty hurts attribute and relationship predictions.
  3. The motivation study only covers 100 coco images. We recommend increasing the probe data set.
Questions for authors

See weaknesses.

1 comment

Weakness 1: Association between image attention and prediction accuracy

Response: We thank the reviewer for this valuable comment. As the reviewer noted, high image-attention scores do not necessarily indicate accurate token predictions. To further examine the effect of image-attention reweighting on generation performance, we extend the evaluation of LaViDa on COCO Caption with k=2k = 2, setting λ=0\lambda = 0. We vary γ\gamma from 0 to 32, where γ=0\gamma = 0 corresponds to confidence-only selection and increasing γ\gamma places greater emphasis on image attention. We additionally evaluate attention-only selection by removing the confidence term from the scoring function, yielding rti=(mti/mtmed)γr_t^i = (m_t^i / m_t^{\mathrm{med}})^\gamma.

γ\gammaCIDEr
0 (confidence only)100.2
1101.5
2103.0
4100.2
898.6
1693.0
3287.0
Confidence removed (attention only)83.2

As shown in the table, CIDEr initially improves as γ\gamma increases from 0 to 2, but decreases consistently beyond this range, with attention-only selection yielding the lowest score. These results show that excessive reliance on image attention can degrade generation quality.

To further quantify the relationship between image-attention scores and token correctness, we randomly mask 60% of the tokens in ground-truth captions of 500 COCO samples and use Lavida to predict the masked tokens. We consider a predicted token correct if it exactly matches the corresponding ground-truth token.

& & & m_t^i/m_t^{\mathrm{med}} & \\\\ & & [0.24,0.81) & [0.81,0.94) & [0.94,1.68] \\\\ \hline & [0.04,0.39) & 25.3 & 27.9 & 25.7 \\\\ c_t^i & [0.39,0.74) & 59.5 & 59.1 & 52.3 \\\\ & [0.74,1.00] & 87.3 & 84.7 & 82.7 \end{array}$$ *Each cell shows the accuracy (%).* As shown in the table, token correctness is more strongly associated with prediction confidence than with the image-attention score. These results reinforce the importance of incorporating confidence into our sampling criterion, rather than relying solely on image attention. We have incorporated the corresponding discussion into Appendix C.3 of the revised manuscript. **Weakness 2:** Effect of similarity regularization on attribute and relational predictions **Response:** We thank the reviewer for raising this concern. We acknowledge that tokens attending to the same image region may convey complementary information, such as an object and its attributes, and that similarity regularization may delay\$^*\$ the decoding of such tokens. However, deferring such tokens is not necessarily a drawback, as it can mitigate prediction inconsistencies arising from parallel decoding. Specifically, in dMLLMs, each masked token is predicted without conditioning on other tokens predicted within the same decoding step. Consequently, simultaneously decoding an object token and its associated attribute tokens may lead to repetition or incorrect word order (e.g., "dog dog" or "dog brown" instead of "brown dog"). In contrast, decoding these tokens in separate steps allows the token decoded first to serve as context for predicting the other token, reducing the risk of such errors. To assess the impact of similarity regularization on fine-grained attribute and relational predictions, we conduct an additional ablation study on DetailCaps by comparing $\lambda = 0$ with the original setting ($\lambda = 3$). DetailCaps evaluates detailed, long-form image captions describing multiple objects, their attributes, and inter-object relations. In addition to the overall CAPTURE score reported in the paper, we report Object, Attribute, and Relation F1 scores, which measure the correspondence between these elements in generated and reference captions. The consistent improvements across all three F1 scores demonstrate that similarity regularization improves captioning performance without compromising attribute and relational predictions. We have incorporated these results into Appendix C.4 of the revised manuscript. | $k$ | $\lambda$ | Object F1 | Attribute F1 | Relation F1 | |:---:|:---:|:---:|:---:|:---:| | 2 | 0 | 61.9 | 54.2 | 53.5 | | | 3 | **62.2** | **54.7** | **53.9** | | 4 | 0 | 54.4 | 44.9 | 49.0 | | | 3 | **59.8** | **53.2** | **53.2** | | 8 | 0 | 34.8 | 24.5 | 28.3 | | | 3 | **46.7** | **38.0** | **42.5** | --- $^*$ Since the number of tokens decoded per step is fixed in our experimental setup, delaying certain tokens does not increase the total number of decoding steps. **Weakness 3:** Increasing the dataset size for the motivation study **Response:** We thank the reviewer for this valuable suggestion. Following the reviewer’s recommendation, we expanded the probe dataset from 100 to 1,500 images, using 500 images each from COCO Captions, TextVQA, and DocVQA. The updated results in Figure 2 confirm that the trends observed in the original study remain consistent with the larger dataset.

Comments on the paper

1 comment
Area_Chair_22rkOpenReview ↗

rebuttal ddl

Dear reviewers, the ICLR deadline is only 4 days away. The authors have already rebuttaled the issues raised. If you still have questions, please participate in the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.