- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes VIVA, a training-free decoding strategy for diffusion multimodal large language models (dMLLMs). The method is motivated by two observations: (1) visually informative tokens tend to be decoded at later stages under confidence-based decoding, and (2) jointly decoding tokens with high image attention leads to larger parallel prediction errors. To address these issues, VIVA incorporates image attention into token selection to prioritize visually informative tokens and introduces hierarchical constraints to distribute visually informative tokens across decoding steps. The method also groups adjacent tokens according to their image-attention distributions to limit the simultaneous decoding of potentially dependent tokens.
Critical strengths
S1. The method demonstrates performance improvements over existing decoding strategies, particularly under highly parallel decoding settings.
S2. VIVA is training-free and introduces relatively small inference overhead, making it applicable to existing dMLLMs without model modifications.
S3. The analysis of decoding trajectories and image attention provides an interesting perspective on token selection in diffusion MLLMs.
Critical weaknesses
W1. Insufficient justification for prioritizing visually informative tokens.
The paper assumes that visually informative tokens should be decoded earlier so that they can provide useful context for subsequent predictions. However, the motivation for this assumption is not sufficiently established. Since visual information is already available throughout the decoding process, it is unclear why committing high-image-attention tokens earlier would necessarily improve prediction quality. Moreover, high image attention does not imply high prediction confidence, and prematurely committing uncertain tokens could potentially introduce errors into subsequent decoding steps.
While the paper demonstrates that visually informative tokens tend to be decoded later, this observation alone does not establish that late decoding is harmful.
W2. Unclear connection between parallel prediction error and actual generation quality.
The paper motivates its method by showing that jointly decoding tokens with high image attention results in larger parallel prediction errors. However, these errors characterize the discrepancy between joint and factorized predictive distributions, rather than actual prediction errors with respect to ground-truth tokens.
A larger distributional discrepancy does not necessarily imply incorrect token predictions or degraded generation quality. Therefore, the paper does not sufficiently establish why tokens with high parallel prediction errors should be decoded separately.
Furthermore, the proposed segmentation assumes that tokens attending to similar visual regions exhibit stronger predictive dependencies. However, the analysis primarily examines image-attention mass rather than attention-distribution similarity, leaving the justification for attention-based segmentation unclear.
W3. Inconsistent improvements across tasks and limited understanding of failure cases.
Although VIVA achieves substantial improvements in several settings, its effectiveness varies considerably across models, tasks, and decoding budgets. For example, the improvements on the understanding benchmarks in Table 1 are less consistent across different backbones, while Table 2 also includes settings in which VIVA underperforms confidence-based decoding or VRCD.
The paper does not sufficiently analyze why the method performs well in some settings but degrades performance in others. A more detailed investigation of these failure cases would help clarify when the proposed strategy is effective and whether its underlying assumptions generalize across different tasks.
W4. Limited justification for the hierarchical attention constraints.
Equation (10) distributes the remaining attention mass according to the number of remaining decoding steps. However, the paper does not sufficiently justify why this allocation is appropriate for minimizing parallel prediction errors.
In addition, although the theoretical analysis provides a mutual-information-based upper bound on dependency error, replacing mutual information with image-attention mass is a heuristic approximation rather than a theoretically guaranteed bound. The connection between the proposed attention-based constraints and actual dependency reduction therefore remains insufficiently validated.
Questions for authors
Q1. Is there qualitative or quantitative evidence that decoding visually informative tokens earlier improves subsequent predictions? In particular, can the authors isolate the benefit of early visual-token decoding from that of reducing simultaneous token selection?
Q2. Why does VIVA show inconsistent improvements across tasks and decoding settings? What are the main causes of the performance degradation observed in some settings in Tables 1 and 2?
Q3. Can the authors provide empirical evidence that image-attention similarity correlates with actual inter-token dependency? How does the proposed attention-based segmentation compare with simpler alternatives, such as fixed-size or position-based grouping?
Q4. Why should larger parallel prediction errors necessarily lead to lower generation accuracy? Can the authors directly analyze the relationship between the proposed dependency-error metric and actual token prediction errors?
Q5. What motivates the allocation rule in Equation (10)? Have the authors compared it against alternative allocation strategies or thresholds?