- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies where attention goes in diffusion multimodal LLMs. In LLaDA-V, a few output tokens ("output sinks") soak up a lot of attention; they move between denoising steps, take more as denoising goes on, and the image gets less. About two-thirds of them are masks, punctuation, or function words. Based on this, the authors propose EPS, which is training-free: at each head and step, find the sinks with an elbow rule, keep the low-semantic ones, and push their logits down relative to an anchor token so the freed attention spreads to the other keys, mostly the image. Results on LLaDA-V, LaViDA, and LaViDA-Dream show less hallucination and gains on several benchmarks.
Critical strengths
The sink analysis for dMLLMs is new as far as I know. What I liked most is that the paper checks its own story: limiting the flow to the image removes about half of the CHAIR_S gain, picking the EPS tokens beats random or mass-matched tokens at a common budget (Table 14), and suppressing content words makes things worse (Table 8). This is what turns a nice observation into something I can believe.
Critical weaknesses
My main concern is how reliable the main numbers are. Many of the gaps to the best baseline in Table 2 are under a point (POPE 0.2-0.4, LaViDA MathVista 0.4), and no repeated runs or variance are reported for any benchmark; App. K itself says the highlighting is rank, not significance. The only justification for alpha = 0.8 is Table 16, run on the same CHAIR and LLaVA-Bench numbers. I also could not tell what the CHAIR sample is: all CHAIR_S values in Table 2 are multiples of 0.5, but Table 13, which shares rows with Table 2, has 26.6, 28.1, 29.6, and the image count is never given.
The second concern is the baselines. OPERA, Farsight, and MCA were built for autoregressive models, and the paper does not say how they were ported to masked diffusion decoding or how their settings were chosen; App. B has nothing on this. Since the headline claim rests on these comparisons, I would appreciate more detail here.
Questions for authors
- How many images are in the CHAIR run, and why do Table 2 and Table 13 have different granularity if they are the same sample?
- How exactly were OPERA, Farsight, and MCA adapted to diffusion decoding, and how were their settings chosen?
- MathVista +5.0 and ScienceQA +5.8 on LLaDA-V are quite big for an attention tweak. With the 128-token limit, could answer length or format be part of it?
5 comments
Response to Reviewer CEve (Part2)
2. Baseline adaptations and settings (W2, Q2)
OPERA, Farsight, and MCA were designed for autoregressive MLLMs. We retain their main mechanisms and adapt them to parallel denoising as follows.
| Method | Original mechanism (AR) | Adaptation for diffusion decoding |
|---|---|---|
| OPERA | Penalizes candidate tokens whose attention concentrates on a summary token. If this concentration persists, rolls back and chooses a different continuation. | Applies the penalty to candidate tokens at each masked position before confidence-based unmasking. Replaces rollback with remasking the tokens decoded after the persistent aggregation position, allowing them to be generated again. |
| Farsight | Uses otherwise masked future positions as attention registers. These absorb probability during softmax but contribute no values to the output; their position-dependent scores also encode positional information. | Adds separate register slots with zero value vectors, since bidirectional attention has no unused future positions. Applies the positional bias to attention from response tokens to image/instruction tokens, leaving attention within the response unchanged. |
| MCA | Replaces 1D raster-scan image position indices with indices derived from 2D Manhattan coordinates, and constructs the corresponding causal mask. | Applies the same position-index reassignment to each image crop grid while retaining the dMLLM's non-causal attention mask. |
For OPERA, we use a window of 32 decoded tokens, attention scale 50, penalty weight 1, and the top 5 candidates per masked position. Remasking is triggered when the aggregation position persists for 15 consecutive denoising steps; the previously selected token is excluded at the reselection position. For Farsight, register scores follow the original position-dependent initialization, while the separate slots and image/instruction bias are our diffusion adaptation. MCA is applied only at inference without retraining, whereas the original MCA-LLaVA is trained with its position scheme.
3. Could answer length or format explain the reasoning gains? (Q3)
We analyzed the outputs of LLaDA-V with and without EPS on both benchmarks.
| Metric (Baseline / EPS) | MathVista | ScienceQA |
|---|---|---|
| Mean response length (tokens) | 97.8 / 96.1 | 70.4 / 69.6 |
| Responses reaching the 128 token limit | 22.4% / 21.7% | 7.9% / 7.6% |
Mean response length changes by less than two tokens, and the fraction reaching the token limit changes by less than one percentage point. These small shifts make length or truncation an unlikely explanation for the 5.0- and 5.8-point gains.
Among the MathVista task types below, the gains are larger for figure question answering and geometry than for math word problems:
| MathVista task type | Gain from EPS (points) |
|---|---|
| Figure question answering | +7.1 |
| Geometry problem solving | +6.3 |
| Math word problems | +1.6 |
This pattern is consistent with improved visual grounding, since figure and geometry questions rely directly on visual information.
The revised paper includes the CHAIR protocol and paired comparisons in Appendix S, the baseline adaptations in Appendix P, and the response-length analysis in Appendix T. Section 4.1 clarifies the CHAIR sample, Table 2 is corrected, and Appendix K specifies how α is used across models and tasks.
Follow-up on Our Rebuttal
Dear Reviewer CEve,
Please forgive this brief follow-up. As the author-reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.
In response to your concerns, we clarified the CHAIR evaluation protocol, including the 200-image sample per run and the distinction between the single-run results in Table 2 and the 11-run averages in Table 13. We also rechecked the experimental records and corrected the CHAIR_I and Recall entries for the baseline and EPS in Table 2. To provide a clearer picture of result reliability, we added run-level CHAIR results and paired 95% confidence intervals, along with a more detailed justification for choosing α = 0.8. In addition, Appendix P now explains how OPERA, Farsight, and MCA were adapted to diffusion decoding, while Appendix T provides response-length, truncation, and task-level analyses to address the possibility that the reasoning improvements are related to output length or format.
We hope these additions address your principal concerns. If you find the new evidence convincing, we would be sincerely grateful if you could reconsider your overall assessment. If any concern remains, even a brief indication would help us provide further clarification promptly.
We appreciate your time and thoughtful review.
Best regards, The Authors
Response to Reviewer CEve (Part1)
1. CHAIR sample, granularity, and the choice of α (W1, Q1)
Sample size and the two tables. Each CHAIR run evaluates 200 COCO images, and all methods within a run use the same 200 images, so comparisons are paired. The two tables report different things:
CHAIR_S changes in 0.5 percentage-point increments for a single 200-image run (Table 2), whereas averaging 11 runs permits values such as 26.6, 28.1, and 29.6 (Table 13).
A correction to Table 2. Your question led us to recheck the records. For Baseline and Full EPS, the CHAIR_S of run 1 equals the 11-run mean (30.0 and 23.0). We mistakenly entered the 11-run means of CHAIR_I and Recall for these two rows of Table 2, instead of their run 1 values. The corrected single-run values are:
All 11 runs. Table R1 lists CHAIR_S for every run and every strategy in Table 13.
Table R1. CHAIR_S (%) across 11 runs of 200 images each (lower is better).
The means reproduce Table 13.
Confidence intervals. Figure 5 reports paired 95% confidence intervals based on Student's t distribution over the run-level differences across the 11 runs:
CI95%=dˉ±t0.975,1011sd,
where dˉ and sd are the mean and sample standard deviation of the paired differences. Figure 5 rounds these values to one decimal.
Table R2. Paired CHAIR_S differences and 95% CIs (percentage points).
The CHAIR_S intervals for comparisons involving EPS exclude zero. The mass-matched versus random comparison is the only one in Table R2 whose interval includes zero.
Choice of α. We use α = 0.8 across the main experiments to balance hallucination, recall, and description quality. Table 16, with the corrected single-run values for α = 0.8, shows the trade-off:
Stronger suppression (α = 0.5) lowers CHAIR_S slightly. However, it increases CHAIR_I and reduces Recall and overall description quality, so we chose the setting that balances these metrics. The ablation uses the reported benchmarks. We then keep α = 0.8 across backbones and tasks, with no per-model or per-benchmark retuning.
We will publicly release all experimental results, data, and scripts, including the image lists for each run, model outputs, evaluation code, and analysis scripts, so that the reported numbers can be independently reproduced.