acceptodds
Under review as a conference paper at ICLR 2027

Regulating Dynamic Attention Allocation in Diffusion Multimodal Large Language Models

Abstract

Diffusion multimodal large language models (dMLLMs) generate text by iteratively denoising a masked response, and at every step, image and output tokens compete for the same softmax attention budget. We find that a small set of output tokens, which we term output sinks, absorbs a disproportionate share of this budget. These sinks shift across positions between steps and claim a growing share of attention as denoising proceeds, at the expense of the image. Such diversion limits the visual evidence available for generation and can induce hallucination. Most sinks are low-semantic tokens (masks, punctuation, and function words), but some are content words. Motivated by these findings, we propose Elbow Proportional Suppression (EPS), a training-free attention intervention. For each head at each step, EPS detects sinks with a parameter-free elbow criterion and attenuates only the low-semantic ones relative to an adaptively selected anchor token via an additive logit shift. We prove that EPS redistributes the suppressed attention proportionally to all remaining keys; empirically, most of it is reallocated to the image. Controlled interventions show that both visual reallocation and the choice of suppressed tokens reduce hallucination. Across three dMLLMs, EPS reduces object hallucination (by up to 8.0 points in CHAIR) and improves description quality and multimodal reasoning, achieving the best results on 24 of 26 metrics against five training-free baselines. Code and analysis tools will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Multimodal Large Language Models, Attention Sinks, Attention Dynamics, Training-Free Inference

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_CEveOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies where attention goes in diffusion multimodal LLMs. In LLaDA-V, a few output tokens ("output sinks") soak up a lot of attention; they move between denoising steps, take more as denoising goes on, and the image gets less. About two-thirds of them are masks, punctuation, or function words. Based on this, the authors propose EPS, which is training-free: at each head and step, find the sinks with an elbow rule, keep the low-semantic ones, and push their logits down relative to an anchor token so the freed attention spreads to the other keys, mostly the image. Results on LLaDA-V, LaViDA, and LaViDA-Dream show less hallucination and gains on several benchmarks.

Critical strengths

The sink analysis for dMLLMs is new as far as I know. What I liked most is that the paper checks its own story: limiting the flow to the image removes about half of the CHAIR_S gain, picking the EPS tokens beats random or mass-matched tokens at a common budget (Table 14), and suppressing content words makes things worse (Table 8). This is what turns a nice observation into something I can believe.

Critical weaknesses

My main concern is how reliable the main numbers are. Many of the gaps to the best baseline in Table 2 are under a point (POPE 0.2-0.4, LaViDA MathVista 0.4), and no repeated runs or variance are reported for any benchmark; App. K itself says the highlighting is rank, not significance. The only justification for alpha = 0.8 is Table 16, run on the same CHAIR and LLaVA-Bench numbers. I also could not tell what the CHAIR sample is: all CHAIR_S values in Table 2 are multiples of 0.5, but Table 13, which shares rows with Table 2, has 26.6, 28.1, 29.6, and the image count is never given.

The second concern is the baselines. OPERA, Farsight, and MCA were built for autoregressive models, and the paper does not say how they were ported to masked diffusion decoding or how their settings were chosen; App. B has nothing on this. Since the headline claim rests on these comparisons, I would appreciate more detail here.

Questions for authors
  1. How many images are in the CHAIR run, and why do Table 2 and Table 13 have different granularity if they are the same sample?
  2. How exactly were OPERA, Farsight, and MCA adapted to diffusion decoding, and how were their settings chosen?
  3. MathVista +5.0 and ScienceQA +5.8 on LLaDA-V are quite big for an attention tweak. With the 128-token limit, could answer length or format be part of it?
5 comments
AuthorsOpenReview ↗

Response to Reviewer CEve (Part1)

1. CHAIR sample, granularity, and the choice of α (W1, Q1)

Sample size and the two tables. Each CHAIR run evaluates 200 COCO images, and all methods within a run use the same 200 images, so comparisons are paired. The two tables report different things:

  • Table 2 reports a single run (run 1).
  • Table 13 reports the mean over 11 runs: run 1 plus 10 additional runs, each on a different set of 200 images (2,200 image evaluations in total).

CHAIR_S changes in 0.5 percentage-point increments for a single 200-image run (Table 2), whereas averaging 11 runs permits values such as 26.6, 28.1, and 29.6 (Table 13).

A correction to Table 2. Your question led us to recheck the records. For Baseline and Full EPS, the CHAIR_S of run 1 equals the 11-run mean (30.0 and 23.0). We mistakenly entered the 11-run means of CHAIR_I and Recall for these two rows of Table 2, instead of their run 1 values. The corrected single-run values are:

MethodCHAIR_S ↓CHAIR_I ↓ (reported → corrected)Recall ↑ (reported → corrected)
Baseline30.07.6 → 7.566.5 → 66.3
Full EPS23.06.0 → 6.169.2 → 69.0

All 11 runs. Table R1 lists CHAIR_S for every run and every strategy in Table 13.

Table R1. CHAIR_S (%) across 11 runs of 200 images each (lower is better).

RunBaselineImage constrained EPSFull EPSEPS commonMass matchedRandom
130.029.023.029.530.530.5
236.031.522.032.534.034.5
330.523.018.023.026.531.0
428.522.523.522.526.031.5
531.032.526.526.027.030.0
627.528.523.525.524.527.0
728.528.024.028.029.031.5
835.527.030.032.536.032.5
933.523.018.524.528.530.5
1026.524.522.525.524.523.5
1122.523.021.522.023.023.0
Mean30.0026.5923.0026.5028.1429.59

The means reproduce Table 13.

Confidence intervals. Figure 5 reports paired 95% confidence intervals based on Student's t distribution over the run-level differences across the 11 runs:

CI95%=dˉ±t0.975, 10 sd11,\mathrm{CI}_{95\%} = \bar d \pm t_{0.975,\,10}\,\frac{s_d}{\sqrt{11}},

where dˉ\bar d and sds_d are the mean and sample standard deviation of the paired differences. Figure 5 rounds these values to one decimal.

Table R2. Paired CHAIR_S differences and 95% CIs (percentage points).

ComparisonMean difference95% CI
Full EPS − Baseline−7.00[−10.12, −3.88]
Image constrained − Baseline−3.41[−6.23, −0.59]
Full EPS − Image constrained−3.59[−5.95, −1.24]
Full EPS − EPS common−3.50[−5.78, −1.22]
EPS common − Random−3.09[−5.38, −0.80]
EPS common − Mass matched−1.64[−2.83, −0.44]
Mass matched − Random−1.45[−3.18, 0.27]

The CHAIR_S intervals for comparisons involving EPS exclude zero. The mass-matched versus random comparison is the only one in Table R2 whose interval includes zero.

Choice of α. We use α = 0.8 across the main experiments to balance hallucination, recall, and description quality. Table 16, with the corrected single-run values for α = 0.8, shows the trade-off:

αCHAIR_S ↓CHAIR_I ↓Recall ↑LLaVA-Bench All ↑
0.522.07.268.870.3
0.823.06.169.072.1

Stronger suppression (α = 0.5) lowers CHAIR_S slightly. However, it increases CHAIR_I and reduces Recall and overall description quality, so we chose the setting that balances these metrics. The ablation uses the reported benchmarks. We then keep α = 0.8 across backbones and tasks, with no per-model or per-benchmark retuning.

We will publicly release all experimental results, data, and scripts, including the image lists for each run, model outputs, evaluation code, and analysis scripts, so that the reported numbers can be independently reproduced.

AuthorsOpenReview ↗

Response to Reviewer CEve (Part2)

2. Baseline adaptations and settings (W2, Q2)

OPERA, Farsight, and MCA were designed for autoregressive MLLMs. We retain their main mechanisms and adapt them to parallel denoising as follows.

MethodOriginal mechanism (AR)Adaptation for diffusion decoding
OPERAPenalizes candidate tokens whose attention concentrates on a summary token. If this concentration persists, rolls back and chooses a different continuation.Applies the penalty to candidate tokens at each masked position before confidence-based unmasking. Replaces rollback with remasking the tokens decoded after the persistent aggregation position, allowing them to be generated again.
FarsightUses otherwise masked future positions as attention registers. These absorb probability during softmax but contribute no values to the output; their position-dependent scores also encode positional information.Adds separate register slots with zero value vectors, since bidirectional attention has no unused future positions. Applies the positional bias to attention from response tokens to image/instruction tokens, leaving attention within the response unchanged.
MCAReplaces 1D raster-scan image position indices with indices derived from 2D Manhattan coordinates, and constructs the corresponding causal mask.Applies the same position-index reassignment to each image crop grid while retaining the dMLLM's non-causal attention mask.

For OPERA, we use a window of 32 decoded tokens, attention scale 50, penalty weight 1, and the top 5 candidates per masked position. Remasking is triggered when the aggregation position persists for 15 consecutive denoising steps; the previously selected token is excluded at the reselection position. For Farsight, register scores follow the original position-dependent initialization, while the separate slots and image/instruction bias are our diffusion adaptation. MCA is applied only at inference without retraining, whereas the original MCA-LLaVA is trained with its position scheme.

3. Could answer length or format explain the reasoning gains? (Q3)

We analyzed the outputs of LLaDA-V with and without EPS on both benchmarks.

Metric (Baseline / EPS)MathVistaScienceQA
Mean response length (tokens)97.8 / 96.170.4 / 69.6
Responses reaching the 128 token limit22.4% / 21.7%7.9% / 7.6%

Mean response length changes by less than two tokens, and the fraction reaching the token limit changes by less than one percentage point. These small shifts make length or truncation an unlikely explanation for the 5.0- and 5.8-point gains.

Among the MathVista task types below, the gains are larger for figure question answering and geometry than for math word problems:

MathVista task typeGain from EPS (points)
Figure question answering+7.1
Geometry problem solving+6.3
Math word problems+1.6

This pattern is consistent with improved visual grounding, since figure and geometry questions rely directly on visual information.

The revised paper includes the CHAIR protocol and paired comparisons in Appendix S, the baseline adaptations in Appendix P, and the response-length analysis in Appendix T. Section 4.1 clarifies the CHAIR sample, Table 2 is corrected, and Appendix K specifies how α is used across models and tasks.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer CEve,

Please forgive this brief follow-up. As the author-reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.

In response to your concerns, we clarified the CHAIR evaluation protocol, including the 200-image sample per run and the distinction between the single-run results in Table 2 and the 11-run averages in Table 13. We also rechecked the experimental records and corrected the CHAIR_I and Recall entries for the baseline and EPS in Table 2. To provide a clearer picture of result reliability, we added run-level CHAIR results and paired 95% confidence intervals, along with a more detailed justification for choosing α = 0.8. In addition, Appendix P now explains how OPERA, Farsight, and MCA were adapted to diffusion decoding, while Appendix T provides response-length, truncation, and task-level analyses to address the possibility that the reasoning improvements are related to output length or format.

We hope these additions address your principal concerns. If you find the new evidence convincing, we would be sincerely grateful if you could reconsider your overall assessment. If any concern remains, even a brief indication would help us provide further clarification promptly.

We appreciate your time and thoughtful review.

Best regards, The Authors

Reviewer_uLjDOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1(−1)
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies attention allocation in diffusion multimodal large language models (dMLLMs), where image and output tokens compete for a shared softmax attention budget during iterative denoising. The authors identify a small set of output positions, termed output sinks, that receive a disproportionate share of attention. They report that these positions can move across denoising steps and that the aggregate attention share allocated to the detected elbow set increases as denoising proceeds, while the image-region share decreases.

Critical strengths
  1. The competition between image keys and output keys in a shared attention normalization is a meaningful property of diffusion-based multimodal generation. Studying this mechanism is a useful direction, particularly because output positions are processed bidirectionally and repeatedly during denoising.

  2. The method does not require fine-tuning or additional model forward passes and preserves the baseline denoising schedule. This makes the approach potentially useful as an inference-time intervention, subject to the reported implementation overhead.

Critical weaknesses
  1. The main attention population study uses LLaDA-V, a fixed image-captioning prompt, 500 COCO validation images, 128 output slots/denoising steps, and only layers 8, 16, 24, and 32. The large number of recorded attention entries is generated by repeated measurements over heads, layers, steps, and images; it should not be interpreted as an equally large number of independent examples. Although EPS is evaluated on additional backbones and tasks, the paper does not establish that the same sink dynamics and token-composition patterns occur across those settings.

  2. The appendix distinguishes the robust concentration-event criterion from geometric elbow detection and explicitly notes that the elbow rule can return a non-empty set below the concentration threshold. This raises a practical question: when the attention distribution is flat or only weakly concentrated, does EPS still suppress some low-semantic positions? If so, the procedure may act on arbitrary candidates rather than genuine sinks. The paper should clarify whether the robust concentration criterion gates the intervention in the actual algorithm and quantify the frequency and consequences of suppression in no-sink/weak-sink cases.

  3. EPS requires attention-probability access and additional reductions, sorting, and logit updates. The appendix also notes the use of eager attention for analysis. The reported latency overhead is useful, but it may not characterize long sequences, larger visual inputs, or production implementations using fused attention kernels. The paper should avoid implying that the measured overhead is universal.

Questions for authors
  1. Does the implementation suppress tokens whenever the geometric elbow rule returns a non-empty set, or is suppression gated by the robust concentration-event criterion?

  2. Can the authors provide the same attention-population analysis for at least one additional dMLLM backbone (e.g., LaViDA) and for a non-captioning task?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uLjD (Part1)

Thank you for your time and effort in reviewing our paper. We respond to your concerns point by point below.

1. Generality across backbones and tasks (W1, Q2)

The 8.2M attention records are repeated measurements from 500 images. Section 2 averages records within each image and phase before averaging across images, so the image is the unit of analysis.

Additional backbones. We repeated the population analysis on LaViDA, as requested, and MMaDA. LaViDA has its own visual pathway and multimodal training, while MMaDA uses discrete visual tokens. We use the same 2,000 COCO val2014 images and analysis protocol across models, with model-specific token IDs and layers sampled at one quarter, one half, three quarters, and full depth.

Table R1. Population statistics across backbones. Values are early phase → late phase. The final column reports mean low-semantic share of elbow-set mass, with the range across strata.

ModelEvent freq.Top 1 (uniform 0.0078)Switch rateMigrationElbow set shareImage shareLow semantic share of E mass
LLaDA-V0.955 → 0.9380.088 → 0.0710.229 → 0.1430.065 → 0.0390.161 → 0.1960.397 → 0.3410.643 (0.481 to 0.832)
LaViDA0.948 → 0.9290.085 → 0.0690.216 → 0.1390.061 → 0.0380.158 → 0.1900.391 → 0.3440.657 (0.493 to 0.837)
MMaDA0.721 → 0.6950.060 → 0.0560.041 → 0.0290.018 → 0.0140.108 → 0.1320.413 → 0.3760.758 (0.624 to 0.879)

The same patterns appear in both additional models: output attention is concentrated, dominant sink positions change during denoising, elbow-set share rises as image share falls, and low-semantic tokens receive most of the elbow-set mass. MMaDA has lower sink mobility, showing that the strength of these effects depends on the model.

A non-captioning task. We also analyzed LLaDA-V on ScienceQA questions containing an image, retaining the 128-token budget. This tests the same attention statistics under question answering rather than the fixed description prompt.

Table R2. Captioning and ScienceQA. Values are early phase → late phase.

TaskEvent freq.Top 1Elbow set shareImage shareSwitch rateLow semantic share of E mass
Captioning (COCO)0.955 → 0.9380.088 → 0.0710.161 → 0.1960.397 → 0.3410.229 → 0.1430.643
ScienceQA0.947 → 0.9290.083 → 0.0690.153 → 0.1840.389 → 0.3470.208 → 0.1260.714

ScienceQA shows the same concentration and declining mobility, with rising elbow-set share alongside declining image share. Low-semantic tokens also account for most of the elbow-set mass in this task.

AuthorsOpenReview ↗

Response to Reviewer uLjD (Part2)

2. Suppression without a robust concentration event (W2, Q1)

The robust criterion does not gate EPS. Algorithm 1 forms the eligible set S=E∩LtS = E \cap L_t and applies the row-wise update when it is nonempty; Equation 11 is a descriptive statistic only. For each row, EPS adds log⁡γi\log\gamma_i to the selected logits, where γi=min⁡{1,αPia/(mi+ϵ)}\gamma_i = \min\{1,\alpha P_{ia}/(m_i+\epsilon)\}. A row with γi=1\gamma_i=1 remains unchanged.

A flat or weakly concentrated distribution can therefore yield eligible targets. When their probabilities are close to the anchor's, γi≈α=0.8\gamma_i\approx\alpha=0.8, so the adjustment is mild. Table R3 measures how often this occurs and how much attention is removed.

Table R3. Robust and weak cases along the EPS trajectory. We use the Table 10 setting (LLaDA-V, 200 COCO images, layers 8/16/24/32, 128 steps, α=0.8\alpha=0.8). Cases are classified before the update, with sample-first averaging. A robust event means Crobust>3.5C_{\mathrm{robust}}>3.5; a weak case has no robust event but a nonempty SS.

CaseShare of records (%)Removed mass per record
Robust event, S nonempty85.80.1080
Robust event, S empty0.80
No robust event, S nonempty (weak case)12.00.0099
No robust event, S empty1.40

Weak cases account for 1.3% of all removed mass. Their updates are smaller, and 22.8% of their query rows remain unchanged. Eligibility still restricts suppression to low-semantic positions.

Quality consequence. We compared the original EPS with a variant gated by the robust criterion, using the single CHAIR run of Table 2.

Table R4. Gated and ungated EPS.

Variant (LLaDA-V, α = 0.8)CHAIR_S ↓CHAIR_I ↓Recall ↑
Baseline30.07.566.3
EPS gated by the robust criterion (C_robust > 3.5)23.56.368.8
EPS (ungated, as in the paper)23.06.169.0

Gated and ungated EPS give similar scores, consistent with the small contribution of weak cases to removed mass. We retain the ungated rule, which avoids an additional threshold.

3. Latency and implementation cost (W3)

The reported CHAIR overheads (7.6% on LLaDA-V and 6.5% on LaViDA) apply to eager attention, 128 output tokens, and each model's default image resolution. We have made this scope explicit in Section 4.6.

EPS requires attention probabilities for column-mass aggregation, which standard fused kernels such as FlashAttention do not materialize. The latency measurements use eager attention. Attention-materialization costs increase with sequence length and the number of visual tokens. The additional operations cost O(ML+Nlog⁡N)O(ML+N\log N) per head and step (Appendix N); actual runtime also depends on sequence length, image-token count, and the attention implementation.

We have added the cross-backbone and ScienceQA analyses to Appendix O, the weak-case and gating experiments to Appendix Q.3, and the implementation details to Appendix N.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer uLjD,

Please forgive this brief follow-up. As the author-reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.

We have added several analyses to address your concerns about generalizability, suppression in weakly concentrated cases, and computational overhead. First, we expanded the population study to 2,000 images and we also added cross-backbone analyses on LaViDA, Dimple, and MMaDA, as well as a non-captioning ScienceQA analysis. Second, we clarified that the robust concentration criterion does not gate EPS and quantified the effect of weak-sink cases. Although these cases account for 12.0% of records, they contribute only 1.3% of the total removed attention mass. Gated and ungated variants also produce similar downstream results. Finally, we clarified the scope of our latency measurements and discussed the additional cost of attention probability materialization, particularly for longer sequences, larger visual inputs, and fused attention implementations.

We hope these additional experiments and clarifications address your principal concerns. If you find the new evidence convincing, we would be sincerely grateful if you could reconsider your overall assessment. If any concern remains, even a brief indication would help us provide further clarification promptly.

We appreciate your time and thoughtful review.

Best regards, The Authors

Reviewer_nMyUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates dynamic output attention sinks in diffusion multimodal language models and proposes Elbow Proportional Suppression (EPS), a training-free method that detects high-attention output tokens and suppresses low-semantic ones through anchor-calibrated attention logit adjustments. Experiments on three dMLLMs report improvements in hallucination, visual description, and multimodal reasoning benchmarks.

Critical strengths

The paper is clearly presented and provides extensive evaluations across multiple diffusion MLLMs and benchmarks. The reported improvements in hallucination metrics are encouraging.

Critical weaknesses
  1. Limited novelty relative to OPERA (Huang et al., CVPR 2024). OPERA already identified excessive attention aggregation on low-information summary tokens, including punctuation, as a cause of visual hallucination and proposed inference-time penalties. EPS follows essentially the same motivation, primarily changing the suppression targets and intervention mechanism. Its elbow detection and anchor-based scaling do not establish a substantially new conceptual contribution.

  2. Insufficient diffusion-specific methodological contribution. EPS does not explicitly exploit denoising timesteps, masking dynamics, token uncertainty, or diffusion transitions. Its attention suppression rule appears applicable to generic Transformers. Moreover, the paper lacks direct comparisons against simple low-semantic token suppression or fixed attention penalties, making it difficult to attribute improvements to the proposed components rather than generic attention reweighting.

Questions for authors
  1. What is the essential methodological novelty beyond OPERA, apart from modifying the intervention location and selected token categories?
  2. Can the authors compare EPS against fixed penalties on low-semantic tokens and Top-K attention sink suppression under matched suppression budgets? 3.Most important thing afterall is the proof of connection between sink and hallucination, what's the ratio of sink among all, and hallucination among sink, could author provide visualization of sinked attention map (include instruction/image token and output token) and according halluciantion ouput text aside.
3 comments
AuthorsOpenReview ↗

Response to Reviewer nMyU (Part1)

1. Novelty beyond OPERA (W1, Q1)

OPERA is the closest prior work, and both methods aim to reduce harmful attention aggregation. EPS addresses the changing attention pattern during diffusion decoding. The main differences are as follows.

  • Denoising dynamics. OPERA analyzes aggregation in a causal window of generated tokens. EPS recomputes targets at every head and denoising step over both masked and filled positions. Dominant sinks switch in 13.5% to 23.7% of adjacent steps, and masks receive up to 71.4% of elbow-set mass (Section 2; Figure 8).
  • Intervention. Original OPERA penalizes beam-candidate scores and uses rollback. EPS directly shifts attention logits, with guaranteed proportional redistribution to all unsuppressed keys, including image keys (Proposition 3.1), while preserving the decoding schedule. This changes the visual information available within the forward pass before token selection.
  • Target selection. EPS filters the detected sinks by token identity and protects content words. In Table 8, suppressing content sinks worsens CHAIR_S from 33.5 to 35.0; suppressing the whole elbow set gives 29.0, versus 27.0 for low-semantic sinks alone.
  • Adaptive suppression. The elbow selects a variable number of targets; the anchor calibrates each query row's suppression, yielding Pij′≤αPia′P'_{ij} \leq \alpha P'_{ia} for suppressed keys under the exact-arithmetic update in Proposition 3.1. Together, these two components set the target count and suppression strength from the current attention state.

For the evaluated diffusion variant of OPERA, candidate penalties are applied at each masked position before confidence-based unmasking, and rollback becomes remasking tokens decoded after a persistent aggregation position.

2. Diffusion specificity and simple baselines (W2, Q2)

EPS adapts to the changing masked sequence. It recomputes sink membership and token eligibility after each denoising update and acts on both masked and filled queries. The probability update itself is general, but its targets depend on the current mask pattern and decoded token identities. It preserves the schedule and confidence rule. In the single CHAIR run used for Table 2, adding EPS lowers CHAIR_S from 24.0 to 22.5 with AdaBlock and from 25.0 to 22.0 with Wavefront, showing compatibility with schedule methods.

Comparison with simpler suppression rules. We compare fixed penalties on all low-semantic positions LtL_t and Top-K suppression under the common-budget procedure of Appendix I, retaining about 14% of the full EPS dose. Matching the removal budget tests whether target selection helps beyond simply suppressing more attention. Values below are means over the 11 runs used in Table 13.

MethodCHAIR_S ↓CHAIR_I ↓Recall ↑
Baseline30.07.666.5
Random non candidates29.67.866.4
Mass matched non candidates28.17.666.9
(a) Fixed penalty on all of L_t28.77.566.8
(b) Top 1, all categories29.67.666.6
(c) Top 5, all categories29.17.666.9
(d) Top 10, all categories30.37.966.0
(e) Top K ∩ L_t, K = 427.57.467.1
(f) EPS, common budget26.57.467.4

The fixed penalty selects by token identity, while category-blind Top-K selects by attention alone. Filtered Top-K and EPS use both criteria and give lower mean CHAIR_S. EPS achieves the lowest mean CHAIR_S, improving on filtered Top-K by 1.0 point without choosing K.

AuthorsOpenReview ↗

Response to Reviewer nMyU (Part2)

3. Connection between sinks and hallucination (Q3)

The proposed link is through visual attention. By absorbing attention that could reach image keys, sinks can weaken the visual evidence available for generating other tokens.

Interventional evidence. Allowing visual reallocation lowers CHAIR_S by a further 3.6 points relative to image-constrained EPS (95% CI [−5.9, −1.2]). At a common removal budget, EPS also outperforms mass-matched targets by 1.6 points (95% CI [−2.8, −0.4]; Figure 5). The first control tests the destination of released attention; the second tests the choice of suppression targets.

Fraction of sink positions. In LLaDA-V, the elbow set occupies 9.2%, 11.6%, and 13.8% of output positions in the early, middle, and late phases, while receiving 15.5%, 18.9%, and 20.3% of the full attention budget.

Hallucination association. At object-mention commitment, hallucinated mentions have higher elbow-set attention (22.4% vs 16.7%) and lower image attention (30.4% vs 37.4%) than grounded mentions, averaged over heads in layers 8, 16, 24, and 32. Across quartiles of elbow-set attention, hallucination rates rise from 3.1% to 5.2%, 8.1%, and 14.0%.

Hallucination among sinks. Hallucinated object mentions account for 1.91% of content positions in the elbow set, versus 0.75% of all content positions (2.55× enrichment). Most content sinks are therefore not hallucinated object mentions; EPS protects them and suppresses only low-semantic sinks.

Visualization. Figure 1 pairs the image and generated descriptions with baseline/EPS attention maps and the allocation across image, prompt, and output regions. Figure 2 additionally labels concentrated columns by token identity at two denoising steps and layers.

We have added the OPERA comparison to Section 3, the baseline adaptations to Appendix P, the suppression and schedule controls to Appendix Q, and the sink–hallucination analysis to Appendix R.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer nMyU,

Please forgive this brief follow-up. As the author-reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.

We have made several additions to address your concerns about the novelty and diffusion-specific contribution of EPS. In particular, we clarified how EPS differs from OPERA through step-wise sink detection over both masked and filled positions, token-identity-based selection, and anchor-calibrated attention redistribution that explicitly benefits image tokens while preserving the decoding schedule. We also added controlled comparisons against fixed low-semantic penalties and Top-K suppression under matched removal budgets, where EPS outperforms the strongest filtered Top-K alternative by 1.0 CHAIR_S point. More importantly, we added further evidence connecting sinks to hallucination: hallucinated object mentions exhibit higher elbow-set attention and lower image attention at commitment, and their occurrence is enriched 2.55 times within the elbow set compared with all content positions.

We hope these additional analyses and controls help clarify both the methodological contribution and its connection to visual hallucination. If you find the new evidence convincing, we would be sincerely grateful if you could reconsider your overall assessment. If any concern remains, we would be happy to provide further clarification.

We appreciate your time and thoughtful review.

Best regards, The Authors

Reviewer_rQCeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors propose a novel method for attention intervention in diffusion language models. In a nutshell, the proposed method identifies that a significant attention probability mass is allocated to insignificant sink tokens (such as punctuation and other low-semantic-content tokens), which forces the model to allocate less attention to important parts of the input, such as images. As a solution, the authors propose the Elbow Proportional Suppression (EPS) method: during inference, the method identifies a special elbow set of tokens and suppresses attention allocated to it by redistributing it to other tokens. The proposed method showed promising results on a number of benchmarks, outperforming other approaches. Namely, the method results in fewer hallucinations, better multimodal reasoning, and improved textual quality.

Critical strengths
  • The paper considers an important and timely problem of attention distribution in diffusion language models, with clear connection to model hallucinations and multimodal reasoning quality
  • The method is logical, and the authors clearly motivate suppressing attention to insignificant sink tokens
  • The empirical validation includes evaluation across multiple benchmarks, showing improvements over existing approaches
Critical weaknesses
  • The paper is not necessarily easy to follow. Some terms used in the paper are not properly explained, which might reduce its reach to a broader audience. For example, the terms “switch rate” and “migration” in the Figure 3 (a) are not formally defined.
  • Section 2 of the paper, which provides the motivation for the method, feels convoluted and hard to follow. The main analysis is performed on only 500 image samples, while the sample size could be much larger.
  • While the experiments show that EPS improves the capabilities of dMLLMs, the analysis in Section 2 and the experimental evaluation could also be extended to a broader range of models and architectures, such as Dimple, MMaDA, and Omni-Diffusion.
Questions for authors
  • Could the authors formally define “switch rate” and “migration” in Figure 3(a), and ensure that all terms used in this section are properly defined?
  • Can the analysis in Section 2 be performed on a larger dataset and a broader range of architectures?
3 comments
AuthorsOpenReview ↗

Response to Reviewer rQCe (Part1)

Thank you for these suggestions. We address the definitions, motivation, and additional experiments below.

1. Terminology in Figure 3(a) (W1, Q1)

For a fixed image, layer, and head, let jt∗j_t^* be the output position receiving the most attention at denoising step tt, averaged over output queries, and let NN be the number of output positions.

  • Switch rate is the fraction of adjacent steps for which jt∗≠jt−1∗j_t^* \neq j_{t-1}^*, measuring how often the dominant position changes.
  • Migration is the average of ∣jt∗−jt−1∗∣/(N−1)|j_t^* - j_{t-1}^*|/(N-1) over adjacent steps, measuring how far it moves. Steps with no position change contribute zero.

Both metrics track output positions, so a mask becoming a decoded token at the same position does not itself count as a switch. The first step is excluded because it has no predecessor.

We have added these definitions to Section 2.2 and the Figure 3 caption. We have also checked terminology throughout the paper and clarified the following terms at first use:

  • Concentration event (Section 2.1; Appendix C) occurs when the largest received attention mass exceeds the median by more than 3.5 times the robust scale, computed as 1.4826 times the median absolute deviation (Eq. 11).
  • Event frequency (Section 2.1) is the fraction of head–layer–step records with a concentration event, averaged within each image and phase, then across images.
  • Prevalence (Section 2.1) is the fraction of images with at least one concentration event in a phase.
  • Low-semantic eligibility (Section 3.2) restricts suppression to masks, end-of-sequence tokens, punctuation, and listed function words within the detected elbow set; content tokens are excluded.

2. Motivation and larger-sample analysis (W2, Q2)

The motivation for EPS is that output sinks compete with image tokens for attention. Their positions change during denoising, and their share of attention grows as image share declines. EPS therefore detects its targets again at each step. Since some sinks are content words, EPS targets only low-semantic sinks. The controls in Section 4.4 then test whether the released attention improves visual grounding. We have reorganized Section 2 around concentration, dynamics and allocation, and token composition, with procedural details in Appendix C.

We also expanded the LLaDA-V analysis from 500 to 2,000 COCO val2014 images, using the same settings and averaging within each image and phase before averaging across images. The image remains the unit of analysis; measurements over heads and steps are repeated observations within an image.

Table R1. LLaDA-V population statistics with 500 images (original) and 2,000 images (new).

PhaseImagesEvent freq.Top 1Top 5EntropyElbow set shareImage shareSwitch rateMigration
Early5000.9620.0920.2350.8950.1550.4080.2370.068
Early2,0000.9550.0880.2290.8980.1610.3970.2290.065
Middle5000.9430.0730.2080.9050.1890.3580.1550.040
Middle2,0000.9490.0760.2110.9030.1820.3640.1610.042
Late5000.9460.0730.1970.9080.2030.3340.1350.037
Late2,0000.9380.0710.2010.9110.1960.3410.1430.039

The larger sample preserves the early-to-late trends, with a maximum absolute change of 0.011 across the reported statistics. We have added this analysis to Appendix O.1 (Table 17).

AuthorsOpenReview ↗

Response to Reviewer rQCe (Part2)

3. Additional architectures (W3)

We repeated the Section 2 analysis on Dimple and MMaDA using the same 2,000 images and analysis protocol, with model-specific token IDs and layers sampled at one quarter, one half, three quarters, and full depth.

Dimple changes the language backbone and training recipe, while MMaDA uses discrete visual tokens. These models test whether the observed pattern extends beyond LLaDA-V’s particular architecture.

Table R2. Section 2 statistics across architectures (2,000 images; early phase → late phase). The last column gives the mean low-semantic share of elbow set mass, with its range across strata.

ModelEvent freq.Top 1 (uniform 0.0078)Switch rateMigrationElbow set shareImage shareLow semantic share of E mass
LLaDA-V0.955 → 0.9380.088 → 0.0710.229 → 0.1430.065 → 0.0390.161 → 0.1960.397 → 0.3410.643 (0.481 to 0.832)
Dimple0.861 → 0.8240.074 → 0.0630.286 → 0.0960.051 → 0.0210.147 → 0.1760.382 → 0.3370.681 (0.521 to 0.812)
MMaDA0.721 → 0.6950.060 → 0.0560.041 → 0.0290.018 → 0.0140.108 → 0.1320.413 → 0.3760.758 (0.624 to 0.879)

Both models show the same trends: dominant positions become more stable, elbow-set share rises as image share falls, and low-semantic tokens receive most of the elbow-set mass. These results concern attention dynamics. The downstream EPS results cover LLaDA-V, LaViDA, and LaViDA-Dream (Tables 2 and 3).

We have added the cross-backbone analysis to Appendix O.2 (Table 18).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer rQCe,

Please forgive this brief follow-up. As the author-reviewer discussion period approaches its close, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.

Following your suggestions, we have made the presentation more explicit by formally defining switch rate and migration in Section 2.2 and clarifying the terminology used throughout the paper. We also reorganized the motivation in Section 2 around attention concentration, denoising dynamics, and token composition to make the connection to EPS easier to follow. Regarding the experimental scope, we expanded the LLaDA-V population analysis from 500 to 2,000 COCO images, with the main trends remaining consistent. We further extended the analysis to three additional backbones, LaViDA, Dimple, and MMaDA, using the same 2,000-image setting. Finally, we added an analysis on ScienceQA to examine whether the observed attention patterns also hold beyond image captioning. These results are now reported in Appendix O.

We hope these additions address your concerns about clarity, sample size, and generalizability. If you find the expanded analysis convincing, we would be sincerely grateful if you could reconsider your overall assessment. If any concern remains, even a brief indication would help us address it promptly.

We appreciate your time and thoughtful review

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal and Revision Summary

Dear SAC and AC,

Thank you for overseeing the review of our submission (Paper #14031). We sincerely appreciate the time and effort of all reviewers and the AC in managing this process.

We have carefully addressed all technical concerns raised during the review and rebuttal phase. Below, we summarize the status of each reviewer’s feedback and respectfully highlight concerns regarding engagement and consistency.

✅ 1. Reviewer rQCe — Clarity, sample size, and coverage

  • Presentation. We defined switch rate and migration and reorganized Section 2 to clarify the motivation for EPS.
  • Larger study. Expanding from 500 to 2,000 COCO images preserves the main trends, with a maximum absolute change of 0.011 across reported statistics (Appendix O.1).
  • Generality. Additional attention analyses on LaViDA, Dimple, MMaDA, and ScienceQA show consistent allocation and composition patterns (Appendix O).

✅ 2. Reviewer nMyU — Novelty and the link to hallucination

  • Beyond OPERA. EPS recomputes targets over masked and filled positions at each head and step, protects content tokens, and shifts attention logits using an adaptive anchor. It preserves the decoding schedule and guarantees proportional redistribution to unsuppressed keys (Section 3).
  • Simple baselines. At a common removal budget, mean CHAIR_S over 11 runs is 26.5 for EPS, versus 28.7 for fixed low-semantic penalties and 27.5 for low-semantic Top-K. EPS also improves AdaBlock and Wavefront (Appendix Q).
  • Mechanism evidence. Visual reallocation lowers CHAIR_S by 3.59 points versus image-constrained EPS (paired 95% CI [−5.95, −1.24]). At a common budget, EPS beats mass-matched targets by 1.64 points (CI [−2.83, −0.44]). Appendix R adds sink–hallucination statistics; Figures 1 and 2 provide visualizations.

✅ 3. Reviewer uLjD — Generality, weak cases, and cost

  • Analysis unit. We clarified that images are the analysis units, with records averaged within each image and phase, and extended the study across backbones and to ScienceQA.
  • Weak cases. The robust concentration criterion does not gate EPS. Weak cases account for 12.0% of records but only 1.3% of removed mass. Gated and ungated EPS yield CHAIR_S of 23.5 and 23.0 (Appendix Q.3).
  • Runtime scope. The 7.6% and 6.5% overheads apply to eager attention, 128 output tokens, and default image resolutions. We discuss attention materialization and scaling costs (Section 4.6; Appendix N).

The reviewer thanked us for the additional experiments and responses and raised rating from 1 to 2.

✅ 4. Reviewer CEve — Reliability, baselines, and reasoning gains

  • Protocol and uncertainty. Appendix S distinguishes the single 200-image run from the 11-run means and reports run-level scores and paired CIs. Mean CHAIR_S falls from 30.0 to 23.0 with EPS (difference −7.00, 95% CI [−10.12, −3.88]).
  • Baseline details. Appendix P documents the diffusion adaptations of OPERA, Farsight, and MCA. Appendix K explains α = 0.8, held fixed across models and tasks. Other benchmarks retain descriptive rankings from single runs.
  • Length checks. On MathVista and ScienceQA, mean response length changes by fewer than two tokens and truncation by less than one percentage point, alongside gains of 5.0 and 5.8 points (Appendix T).

The reviewer welcomed the confidence intervals and raised rating from 3 to 4 (Clear acceptance), with soundness 3 (no substantive concerns).

Discussion status and request

Two reviewers who have responded to our rebuttal have raised their ratings. Reviewer CEve increased rating from 3 to 4 (Clear acceptance), and reviewer uLjD increased rating from 1 to 2. We have not yet received follow-up feedback from nMyU or rQCe after our rebuttal and reminders.

We hope the added experiments and revisions address the concerns raised in the reviews. We would appreciate your consideration of these updates when making the final decision. Thank you for your time and for helping with the discussion. The code and data will be fully released upon acceptance.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.