- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces the Visual Information-Guided Sampler (VIG-Sampler), a training-free parallel decoding strategy for diffusion Multimodal Large Language Models (dMLLMs). To address the issue of conventional samplers prematurely predicting semantically uninformative tokens (such as punctuation or stop words) based on text-driven certainty, the authors propose leveraging token-to-image attention weights readily available during the decoding process. VIG-Sampler modifies token selection in two ways: first, it reweights token confidence based on image-attention mass to promote visual grounding; second, it introduces a similarity penalty to discourage the concurrent selection of tokens with highly overlapping image-attention distributions, thereby maximizing information gain.
Critical strengths
-
The proposed method elegantly circumvents the severe computational bottleneck of the strong Info-Gain baseline, which requires multiple expensive forward passes to evaluate token combinations. By "free-riding" on the naturally computed self-attention matrices during a single forward pass, VIG-Sampler achieves superior parallel decoding performance with an inference time almost identical to the simplest confidence-based sampling.
-
The preliminary analyses effectively bridge the gap between empirical phenomena and the proposed solution. The authors convincingly demonstrate that a token's image-attention mass correlates with its reliance on visual evidence, and that high attention similarity correlates with redundant information.
-
The authors test the method across a diverse set of multimodal tasks and different model architectures. The consistent performance gains, particularly the finding that VIG-Sampler with can match or surpass conventional samplers at with fewer decoding steps, highlight its effectiveness.
Critical weaknesses
-
While highly effective in practice, the core algorithmic contribution is essentially a combination of existing heuristics. Utilizing attention maps as a proxy for feature importance or visual grounding is a standard interpretability practice. Similarly, penalizing pairwise similarity to reduce redundancy is a well-known diversity-promoting technique (akin to Maximal Marginal Relevance). The novelty lies strictly in its application to dMLLM decoding schedules rather than representing a fundamental methodological innovation.
-
The paper asserts that penalizing attention similarity increases sequence-level "information gain," which has a strict entropy-based definition in Equation 4. However, the connection between the cosine similarity of attention vectors and rigorous sequence entropy reduction is established solely through an empirical observation (Figure 2). The paper lacks a theoretical derivation or proof to explain why lower attention similarity bounds or mathematically approximates the reduction in predictive entropy.
Questions for authors
see weakness
Weakness 1: Novelty beyond existing heuristics
Response: We appreciate the reviewer's perspective and acknowledge that prior research has explored the use of attention maps and similarity-based penalties. We would like to clarify that VIG-Sampler's novelty lies not in the use of these concepts per se, but in the new empirical insights and their translation into a decoding strategy that addresses the limitations of existing dMLLM samplers.
Specifically, to the best of our knowledge, we are the first to investigate the relationship between image attention and the informativeness of decoded tokens for subsequent predictions in dMLLM decoding. Our analysis reveals that (i) tokens with higher image-attention mass are more likely to be visually grounded and yield greater information gain, and (ii) tokens with similar image-attention patterns tend to exhibit greater information redundancy when decoded jointly. These findings naturally motivate us to adopt image-attention-based score reweighting and image-attention similarity regularization as two complementary design choices aimed at increasing set-level information gain.
We believe that our findings and their instantiation in VIG-Sampler, supported by extensive experimental validation, constitute a meaningful contribution to research on visually grounded parallel decoding for dMLLMs.
Weakness 2: Theoretical support for the connection between attention similarity and sequence-level information gain
Response: We appreciate the reviewer’s valuable comment that the connection between attention similarity and sequence-level information gain could be better substantiated. To address this concern, we complement our empirical findings with an information-theoretic explanation. We first discuss the potential information redundancy among tokens with similar image-attention patterns, and then explain its effect on sequence-level information gain.
Prior studies on attention in Transformer-based models have observed that tokens with similar attention patterns tend to have similar representations, i.e., similar hidden states [1, 2]. These findings support the assumption that two tokens with similar image attention may convey overlapping visual information and provide redundant context for subsequent predictions. To connect this potential redundancy to reductions in sequence-level information gain, we first examine its effect at an individual masked position. When the two masked positions in S=i,j are decoded in parallel, the expected token-level information gain at a remaining masked position k∈Mt(S) is expressed as the mutual information
I(yk;yi,yj)=H(yk)−H(yk∣yi,yj).
By the chain rule [3], this gain can be decomposed into the information provided by yi and the additional information provided by yj once yi is known:
\begin{aligned} I(y^k;y^i,y^j) &=I(y^k;y^i) +I(y^k;y^j\mid y^i). \end{aligned}
If the two decoded tokens provide redundant information for predicting yk, the information provided by yj after yi is known can be smaller than that provided by yj alone:
I(yk;yj∣yi)<I(yk;yj).
Combining this condition with the chain rule gives
I(yk;yi,yj)<I(yk;yi)+I(yk;yj).
Thus, informational redundancy between the two tokens leads to their joint information gain being smaller than the sum of their individual gains.
As defined in our manuscript, SIR quantifies the effect of such redundancy on sequence-level information gain, with gains measured as reductions in average marginal predictive entropy. Using this measure, we found that token sets with more similar image-attention patterns had higher SIR values (Figure 2, right), indicating smaller joint information gain relative to the sum of individual gains. Together with the information-theoretic explanation above, this empirical finding provides a rationale for using image-attention similarity as a regularization term in parallel decoding.
We have incorporated this theoretical explanation into Appendix B.2 of the revised manuscript.
[1] Namuk Park et al. “What Do Self-Supervised Vision Transformers Learn?” ICLR, 2023.
[2] Zichen Liu et al. “Token Coordinated Prompt Attention is Needed for Visual Prompting”. ICML, 2025.
[3] Thomas M. Cover and Joy A. Thomas. “Elements of Information Theory”. Wiley, 1991.