- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates causal interventions on "massive values" within the residual streams of three diffusion-based multimodal language models (LaViDa, Dream-VL, and MMaDA), and provides a theoretical account of how these "peaks" form and how they influence the discrete denoising process.
Critical strengths
- The approach to isolating confounding factors is very clean.
- The question of "how an activation is utilized" within diffusion models is significant.
Critical weaknesses
- The number of tasks and the scale of the experiments are too small; each task consists of only 48 inputs, and the results lack statistical significance.
- The intervention that re-selecting and resetting the maximum coordinates for all tokens within a specific region at every layer and every denoising step, is not well-justified. Also in light of Sun et al. (2024), the conclusions reached are overly trivial.
- The observed regional differences are primarily evident in Dream-VL, but for the other two models, the intervention leads to a "consistency collapse." So the findings appear specific to Dream-VL rather than generalizable.
- The "Top" strategy retains coordinates with the largest absolute values, whereas the "Random" strategy (at a 50% retention rate) preserves only about half of the total energy; thus, the superiority of "Top" over "Random" is almost a foregone conclusion.
- The paper lacks any autoregressive control experiments, making it impossible to distinguish which phenomena stem from masked denoising and which are simply inherent properties of Transformers.
Questions for authors
see weakness
5 comments
Response to Reviewer mGsD (2/4)
W2. Rationale for repeated peak selection and contribution beyond Sun et al. (2024).
Repeated selection tests a sustained policy as can generation continue when current peaks in a specified token region are continually suppressed? Representations change across layers and denoising steps, so reselecting the peak implements this target. The resulting scores characterize dependence under that policy; identifying the role of coordinate choice requires additional controls.
We therefore freeze reference coordinates and requested doses from an unmodified trajectory within restricted windows. Compared branches edit the same tokens the same number of times, with actual displacement matched after BF16 writeback. The early and late windows each contain 48 edits per target token. Matching holds between branches within each window; their total displacement can differ across windows.
Our contribution beyond Sun et al. is the controlled separation of representation scale, composition, and downstream use in dMLLMs. Sun et al. [3] established the importance of approximately input-invariant massive activations through zeroing and mean replacement, and analyzed their relation to implicit attention biases. Our norm controls test whether restoring scale also restores function. R/PG controls then distinguish residual additive carry (R) from normalized FFN reading (PG), with the same Q/K/V access rules and final-readout protection. These comparisons refine the functional account of massive activations and are compatible with Sun et al.'s attention-bias interpretation.
ChartQA scores are percentages and differences are pp; COCO uses CIDEr × 100. Each counts independent image clusters.
| Controlled comparison | Model | Dataset | Intervention scope | N | Metrics | Condition A | Score A | Condition B | Score B | B − A [95% CI] |
|---|---|---|---|---|---|---|---|---|---|---|
| Early window | LaViDa | ChartQA | First 6 layers × first 8 steps | 384 | Relaxed Accuracy (%) | Frozen-reference MV | 58.59 | Matched ordinary coordinate | 58.59 | 0.00 [0.00, 0.00] |
| Late window | LaViDa | ChartQA | First 6 layers × last 8 steps | 384 | Relaxed Accuracy (%) | Frozen-reference MV | 5.99 | Matched ordinary coordinate | 57.29 | 51.30 [46.09, 56.51] |
| Representation scale | Dream-VL | ChartQA | Template region | 1,457 | Relaxed Accuracy (%) | Remove MV, restore norm | 0.07 | Preserve direction, reduce norm | 84.83 | 84.76 [82.84, 86.55] |
| Downstream operation | LaViDa | COCO | Template region | 2,000 | CIDEr (×100) | Restore FFN reading (PG) | 0.03 | Restore residual carry (R) | 55.50 | 55.47 [53.42, 57.55] |
| Downstream operation | Dream-VL | COCO | Template region | 2,000 | CIDEr (×100) | Restore FFN reading (PG) | 0.00 | Restore residual carry (R) | 115.11 | 115.11 [111.91, 118.34] |
| Downstream operation | MMaDA | COCO | Template region | 2,000 | CIDEr (×100) | Restore FFN reading (PG) | 0.03 | Restore residual carry (R) | 88.72 | 88.68 [85.93, 91.49] |
The late-window 51.30 pp gap establishes coordinate dependence at matched displacement and editing frequency under the frozen-reference policy. The early window yields no gap, placing the observed dependence in a particular intervention range. In the norm comparison, preserving direction under contraction succeeds while restoring norm after MV removal fails. Across all three COCO template settings, restoring residual carry recovers substantial performance whereas restoring FFN reading alone does not. The contribution is thus a set of experimentally separable functional roles, beyond the observation that deleting large activations can be harmful.
Response to Reviewer mGsD (3/4)
W3. Are regional differences and the resulting conclusions specific to Dream-VL?
Regional heterogeneity also appears in MMaDA in the expanded experiment. On 2,000 independent COCO image clusters per model, MMaDA retains 31.69 CIDEr × 100 after question-region suppression, while its other regional interventions approach zero. LaViDa still reaches the score floor, so its regional ranking remains unresolved. Results use seed 42 and the same continuous zeroing rule; perturbation magnitude varies across regions.
| Model | Native | Zero image MV | Zero question MV | Zero template MV | Zero response MV |
|---|---|---|---|---|---|
| LaViDa | 64.50 | 0.01 | 0.12 | 0.03 | 0.11 |
| Dream-VL | 116.06 | 111.91 | 114.36 | 0.20 | 36.76 |
| MMaDA | 92.03 | 0.03 | 31.69 | 0.17 | 0.44 |
This directly extends the presence of regional differences beyond Dream-VL, while showing that the pattern depends on the model. A separate generalization question concerns the downstream operation supported by the template representation. We test this through R and PG on the same 2,000-image panels, retaining the same Q/K/V access rules and final-readout protection. Scores and differences are CIDEr × 100; pointwise intervals use 10,000 paired image-cluster bootstrap resamples.
| Model | Residual carry R | FFN reading PG | R − PG [95% CI] |
|---|---|---|---|
| LaViDa | 55.50 | 0.03 | 55.47 [53.42, 57.55] |
| Dream-VL | 115.11 | 0.00 | 115.11 [111.91, 118.34] |
| MMaDA | 88.72 | 0.03 | 88.68 [85.93, 91.49] |
All three intervals lie above zero, including LaViDa's. The COCO template carry-versus-reading distinction therefore repeats across all three models. The regional experiments establish heterogeneity in Dream-VL and MMaDA, while the operation experiment provides a common functional contrast even for LaViDa, whose regional scores saturate. The cross-model contribution is this reproducible operation-level result alongside model-dependent regional patterns.
W4. Energy confounding in the Top versus Random comparison.
The fixed-coordinate comparison evaluates two retention policies that differ in both coordinate selection and retained energy. In the original 48-input sustained Dream-VL/ChartQA experiment, Top-50% achieves 85.42% and Random-50% scores zero. Both policies retain the same number of coordinates, but their retained energy differs. We therefore interpret that result as a fixed-budget policy comparison and use the added energy control to quantify the budget distinction directly.
The added experiment quantifies energy concentration and local tolerance. On 384 independent ChartQA inputs with Dream-VL, we apply one retention operation to each of the 128 response positions at a middle layer in the first denoising update. This differs from the original policy, which repeatedly retains coordinates across layers and updates. Top and Random start from the same native state; the matched arms equate retained energy separately for each token. Matching may partially attenuate one boundary coordinate, counted in the nonzero fraction. The Top-50% energy target averages 97.19%; the final column reports the 95th percentile of absolute per-token energy gaps, in pp.
| Retention constraint | Top nonzero (%) | Random nonzero (%) | Retained energy (%)<br>Top / Random | Energy-gap P95 (pp) |
|---|---|---|---|---|
| Same 50% coordinate budget | 50.00 | 50.00 | 97.19 / 49.94 | 65.9324 |
| Match the energy of Top-50% | 50.00 | 95.85 | 97.19 / 97.19 | 0.0250 |
| Same 50% energy target | 0.78 | 50.06 | 50.00 / 50.00 | 0.0439 |
All local arms score 83.07%, including the unmatched Top-50% and Random-50% arms. This measures tolerance of the local edit, with the sustained performance gap already absent before energy matching. At matched 97.19% energy, Top retains 50.00% of coordinates and Random 95.85%, quantifying the concentration expected from magnitude sorting. Determining the source of the original performance gap requires a comparison within the sustained protocol. We therefore retain the Top/Random finding as a fixed-coordinate-budget result, while the norm and R/PG comparisons establish the paper's functional distinctions through separate controls. The added table clarifies the distinction between an energy-concentration property and the functional dependencies tested by the other experiments.
Response to Reviewer mGsD (1/4)
Thank you for examining the reliability and interpretation of our interventions. The expanded experiments strengthen the evidence for the template-MV effect and identify specific functional distinctions. We also clarify what the energy and autoregressive comparisons measure.
Response. The principal template-MV effect now has larger independent samples, paired statistical support, and a three-seed check. Task breadth and sample size address different concerns. The original five tasks cover scientific and scene question answering (ScienceQA-IMG and GQA), text and chart understanding (TextVQA and ChartQA), and open-ended captioning (COCO). We expand three of these existing tasks asScienceQA-IMG, ChartQA, and COCO to test the core template-suppression result across complementary demands: answering a constrained question, extracting chart information, and generating a description.
Each task has a 1,000-image panel nested within a larger panel of 1,965, 1,457, and 2,000 independent image clusters, respectively, all disjoint from the original 48 inputs. Every model–task comparison has a positive native-minus-intervention difference, with pointwise 95% intervals above zero and Holm-adjusted p<0.001 across the nine comparisons per panel. Intervals use 10,000 paired image-cluster bootstrap resamples; full scores appear in Reviewer yAdw W1.
Seeds 42, 123, and 2024 use the same 256 inputs per task. The image remains the independent unit, with its three seed results kept together during resampling. Reductions below are percentage points (pp) for QA and CIDEr × 100 for COCO.
All nine mean reductions have intervals above zero. This establishes the reliability of template suppression across the expanded task panels. The original five-task design defines the breadth of the study; the larger panels deepen evidence for this core phenomenon, and W2–W3 separately report expanded functional controls.