acceptodds
Under review as a conference paper at ICLR 2027

Beyond Magnitude: Massive Activations in Diffusion Multimodal Language Models

Abstract

Diffusion multimodal language models generate responses through repeated masked-token updates, but the computational role of their largest residual activations remains unclear. We investigate these massive values in LaViDa, Dream-VL, and MMaDA through controlled interventions on fixed subsets of five multimodal datasets. Large components occur across image, question, template, and response tokens, yet their functional effects differ across regions, tasks, and models. Peak suppression can disrupt generation, but restoring the original norm does not consistently rescue performance, and retaining the peak alone is insufficient. Amplitude scaling further reveals asymmetric sensitivity. Routing peak access to downstream consumers reveals different dependencies on attention inputs, residual carry, normalized feed-forward reading, and terminal exposure. We develop a conditional theoretical account linking coordinate-concentrated writes and residual propagation to peak formation, and directional perturbations to discrete denoising decisions. These findings characterize massive values through their interaction with surrounding representations and downstream computation, rather than treating magnitude as a direct measure of semantic importance.

Primary area:
causal reasoning
Keywords:
diffusion multimodal large language models, mechanistic interpretability, massive activations, hidden-state representations, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mGsDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates causal interventions on "massive values" within the residual streams of three diffusion-based multimodal language models (LaViDa, Dream-VL, and MMaDA), and provides a theoretical account of how these "peaks" form and how they influence the discrete denoising process.

Critical strengths
  1. The approach to isolating confounding factors is very clean.
  2. The question of "how an activation is utilized" within diffusion models is significant.
Critical weaknesses
  1. The number of tasks and the scale of the experiments are too small; each task consists of only 48 inputs, and the results lack statistical significance.
  2. The intervention that re-selecting and resetting the maximum coordinates for all tokens within a specific region at every layer and every denoising step, is not well-justified. Also in light of Sun et al. (2024), the conclusions reached are overly trivial.
  3. The observed regional differences are primarily evident in Dream-VL, but for the other two models, the intervention leads to a "consistency collapse." So the findings appear specific to Dream-VL rather than generalizable.
  4. The "Top" strategy retains coordinates with the largest absolute values, whereas the "Random" strategy (at a 50% retention rate) preserves only about half of the total energy; thus, the superiority of "Top" over "Random" is almost a foregone conclusion.
  5. The paper lacks any autoregressive control experiments, making it impossible to distinguish which phenomena stem from masked denoising and which are simply inherent properties of Transformers.
Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer mGsD (1/4)

Thank you for examining the reliability and interpretation of our interventions. The expanded experiments strengthen the evidence for the template-MV effect and identify specific functional distinctions. We also clarify what the energy and autoregressive comparisons measure.

W1. Limited task breadth, small experimental scale, and missing statistical significance.

Response.{\color{green}\mathbf{Response.}} The principal template-MV effect now has larger independent samples, paired statistical support, and a three-seed check. Task breadth and sample size address different concerns. The original five tasks cover scientific and scene question answering (ScienceQA-IMG and GQA), text and chart understanding (TextVQA and ChartQA), and open-ended captioning (COCO). We expand three of these existing tasks asScienceQA-IMG, ChartQA, and COCO to test the core template-suppression result across complementary demands: answering a constrained question, extracting chart information, and generating a description.

Each task has a 1,000-image panel nested within a larger panel of 1,965, 1,457, and 2,000 independent image clusters, respectively, all disjoint from the original 48 inputs. Every model–task comparison has a positive native-minus-intervention difference, with pointwise 95% intervals above zero and Holm-adjusted p<0.001p<0.001 across the nine comparisons per panel. Intervals use 10,000 paired image-cluster bootstrap resamples; full scores appear in Reviewer yAdw W1.

Seeds 42, 123, and 2024 use the same 256 inputs per task. The image remains the independent unit, with its three seed results kept together during resampling. Reductions below are percentage points (pp) for QA and CIDEr × 100 for COCO.

ModelDatasetMetricsSeed 42Seed 123Seed 2024Mean reduction [95% CI]
LaViDaScienceQA-IMGAccuracy (pp)74.6174.6174.6174.61 [69.14, 80.08]
LaViDaChartQARelaxed Accuracy (pp)57.4258.9858.9858.46 [52.47, 64.45]
LaViDaCOCOCIDEr (×100)63.5463.5463.5463.54 [58.15, 69.23]
Dream-VLScienceQA-IMGAccuracy (pp)92.9792.9792.9792.97 [89.84, 96.09]
Dream-VLChartQARelaxed Accuracy (pp)83.9884.3884.3884.24 [79.56, 88.54]
Dream-VLCOCOCIDEr (×100)103.88103.88103.88103.88 [95.80, 112.34]
MMaDAScienceQA-IMGAccuracy (pp)56.6456.6456.6456.64 [50.39, 62.50]
MMaDAChartQARelaxed Accuracy (pp)11.7211.7211.7211.72 [7.81, 15.63]
MMaDACOCOCIDEr (×100)88.5588.5588.5588.55 [81.37, 95.85]

All nine mean reductions have intervals above zero. This establishes the reliability of template suppression across the expanded task panels. The original five-task design defines the breadth of the study; the larger panels deepen evidence for this core phenomenon, and W2–W3 separately report expanded functional controls.


AuthorsOpenReview ↗

Response to Reviewer mGsD (2/4)

W2. Rationale for repeated peak selection and contribution beyond Sun et al. (2024).

Response.{\color{green}\mathbf{Response.}} Repeated selection tests a sustained policy as can generation continue when current peaks in a specified token region are continually suppressed? Representations change across layers and denoising steps, so reselecting the peak implements this target. The resulting scores characterize dependence under that policy; identifying the role of coordinate choice requires additional controls.

We therefore freeze reference coordinates and requested doses from an unmodified trajectory within restricted windows. Compared branches edit the same tokens the same number of times, with actual displacement matched after BF16 writeback. The early and late windows each contain 48 edits per target token. Matching holds between branches within each window; their total displacement can differ across windows.

Our contribution beyond Sun et al. is the controlled separation of representation scale, composition, and downstream use in dMLLMs. Sun et al. [3] established the importance of approximately input-invariant massive activations through zeroing and mean replacement, and analyzed their relation to implicit attention biases. Our norm controls test whether restoring scale also restores function. R/PG controls then distinguish residual additive carry (R) from normalized FFN reading (PG), with the same Q/K/V access rules and final-readout protection. These comparisons refine the functional account of massive activations and are compatible with Sun et al.'s attention-bias interpretation.

ChartQA scores are percentages and differences are pp; COCO uses CIDEr × 100. Each NN counts independent image clusters.

Controlled comparisonModelDatasetIntervention scopeNMetricsCondition AScore ACondition BScore BB − A [95% CI]
Early windowLaViDaChartQAFirst 6 layers × first 8 steps384Relaxed Accuracy (%)Frozen-reference MV58.59Matched ordinary coordinate58.590.00 [0.00, 0.00]
Late windowLaViDaChartQAFirst 6 layers × last 8 steps384Relaxed Accuracy (%)Frozen-reference MV5.99Matched ordinary coordinate57.2951.30 [46.09, 56.51]
Representation scaleDream-VLChartQATemplate region1,457Relaxed Accuracy (%)Remove MV, restore norm0.07Preserve direction, reduce norm84.8384.76 [82.84, 86.55]
Downstream operationLaViDaCOCOTemplate region2,000CIDEr (×100)Restore FFN reading (PG)0.03Restore residual carry (R)55.5055.47 [53.42, 57.55]
Downstream operationDream-VLCOCOTemplate region2,000CIDEr (×100)Restore FFN reading (PG)0.00Restore residual carry (R)115.11115.11 [111.91, 118.34]
Downstream operationMMaDACOCOTemplate region2,000CIDEr (×100)Restore FFN reading (PG)0.03Restore residual carry (R)88.7288.68 [85.93, 91.49]

The late-window 51.30 pp gap establishes coordinate dependence at matched displacement and editing frequency under the frozen-reference policy. The early window yields no gap, placing the observed dependence in a particular intervention range. In the norm comparison, preserving direction under contraction succeeds while restoring norm after MV removal fails. Across all three COCO template settings, restoring residual carry recovers substantial performance whereas restoring FFN reading alone does not. The contribution is thus a set of experimentally separable functional roles, beyond the observation that deleting large activations can be harmful.


AuthorsOpenReview ↗

Response to Reviewer mGsD (3/4)

W3. Are regional differences and the resulting conclusions specific to Dream-VL?

Response.{\color{green}\mathbf{Response.}} Regional heterogeneity also appears in MMaDA in the expanded experiment. On 2,000 independent COCO image clusters per model, MMaDA retains 31.69 CIDEr × 100 after question-region suppression, while its other regional interventions approach zero. LaViDa still reaches the score floor, so its regional ranking remains unresolved. Results use seed 42 and the same continuous zeroing rule; perturbation magnitude varies across regions.

ModelNativeZero image MVZero question MVZero template MVZero response MV
LaViDa64.500.010.120.030.11
Dream-VL116.06111.91114.360.2036.76
MMaDA92.030.0331.690.170.44

This directly extends the presence of regional differences beyond Dream-VL, while showing that the pattern depends on the model. A separate generalization question concerns the downstream operation supported by the template representation. We test this through R and PG on the same 2,000-image panels, retaining the same Q/K/V access rules and final-readout protection. Scores and differences are CIDEr × 100; pointwise intervals use 10,000 paired image-cluster bootstrap resamples.

ModelResidual carry RFFN reading PGR − PG [95% CI]
LaViDa55.500.0355.47 [53.42, 57.55]
Dream-VL115.110.00115.11 [111.91, 118.34]
MMaDA88.720.0388.68 [85.93, 91.49]

All three intervals lie above zero, including LaViDa's. The COCO template carry-versus-reading distinction therefore repeats across all three models. The regional experiments establish heterogeneity in Dream-VL and MMaDA, while the operation experiment provides a common functional contrast even for LaViDa, whose regional scores saturate. The cross-model contribution is this reproducible operation-level result alongside model-dependent regional patterns.


W4. Energy confounding in the Top versus Random comparison.

Response.{\color{green}\mathbf{Response.}} The fixed-coordinate comparison evaluates two retention policies that differ in both coordinate selection and retained energy. In the original 48-input sustained Dream-VL/ChartQA experiment, Top-50% achieves 85.42% and Random-50% scores zero. Both policies retain the same number of coordinates, but their retained energy differs. We therefore interpret that result as a fixed-budget policy comparison and use the added energy control to quantify the budget distinction directly.

The added experiment quantifies energy concentration and local tolerance. On 384 independent ChartQA inputs with Dream-VL, we apply one retention operation to each of the 128 response positions at a middle layer in the first denoising update. This differs from the original policy, which repeatedly retains coordinates across layers and updates. Top and Random start from the same native state; the matched arms equate retained energy separately for each token. Matching may partially attenuate one boundary coordinate, counted in the nonzero fraction. The Top-50% energy target averages 97.19%; the final column reports the 95th percentile of absolute per-token energy gaps, in pp.

Retention constraintTop nonzero (%)Random nonzero (%)Retained energy (%)<br>Top / RandomEnergy-gap P95 (pp)
Same 50% coordinate budget50.0050.0097.19 / 49.9465.9324
Match the energy of Top-50%50.0095.8597.19 / 97.190.0250
Same 50% energy target0.7850.0650.00 / 50.000.0439

All local arms score 83.07%, including the unmatched Top-50% and Random-50% arms. This measures tolerance of the local edit, with the sustained performance gap already absent before energy matching. At matched 97.19% energy, Top retains 50.00% of coordinates and Random 95.85%, quantifying the concentration expected from magnitude sorting. Determining the source of the original performance gap requires a comparison within the sustained protocol. We therefore retain the Top/Random finding as a fixed-coordinate-budget result, while the norm and R/PG comparisons establish the paper's functional distinctions through separate controls. The added table clarifies the distinction between an energy-concentration property and the functional dependencies tested by the other experiments.


Reviewer_yAdwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the functional roles of massive residual activations in diffusion multimodal language models (dMLLMs). Through a series of interventions across three models and five datasets, the authors examine how activation magnitude, representation geometry, and downstream computational pathways affect generation. The paper further provides a theoretical analysis connecting massive activation formation, residual propagation, and discrete denoising decisions.

Critical strengths
  1. The paper investigates an interesting and relatively underexplored problem concerning the functional roles of massive activations in diffusion multimodal language models.

  2. The experiments provide a systematic analysis through peak suppression, norm restoration, coordinate retention, amplitude scaling, and downstream pathway interventions, offering multiple perspectives on the functional effects of massive activations.

  3. The study covers three different dMLLMs and multiple multimodal tasks, revealing interesting differences in activation sensitivity across models, token regions, and computational pathways.

Critical weaknesses
  1. The theoretical analysis largely relies on conditional mathematical derivations and previously collected exploratory evidence rather than direct validation of the proposed mechanisms in the main experiments. The experimental scale is limited, with only 48 samples per dataset. Can you provide magnitude-matched perturbation controls or more localized interventions to distinguish the importance of massive activations from general representation disruption?

  2. The interventions repeatedly modify peak coordinates across multiple layers and denoising steps, potentially introducing substantial distribution shifts and cascading errors. The random-coordinate controls are not matched for perturbation magnitude, making it difficult to fully isolate the specific effects of massive activations.

  3. Several interventions cause near-complete performance collapse, but these failures may reflect disrupted generation, premature termination, or empty outputs rather than the loss of task-relevant information. Although the appendix acknowledges these phenomena, the experiments do not sufficiently distinguish their contributions to the observed performance degradation. Can you further separate failures caused by premature termination from those caused by impaired semantic processing?

  4. The authors should clarify whether the identified mechanisms are specific to diffusion-based generation rather than arising from general computational properties of Transformers.

  5. The authors should acknowledge the limitation that some models exhibit extremely low baseline performance on certain tasks, raising concerns about the reliability and generalizability of the findings.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer yAdw (1/4)

Thank you for your careful assessment and for recognizing the design. The additional experiments address the reliability of the effects and distinguish how MV perturbations from how they affect the final answer.

W1. Experimental scale and direct tests of the proposed mechanisms.

Response.{\color{green}\mathbf{Response.}} We added independent evaluations and controlled tests of the representation-use mechanisms in Section 5.2. These test whether preserving numerical properties preserves the computations needed for generation. Section 5.1 provides a conditional formation account supported by the original writer measurements at the inspected nodes. W2 supplies magnitude-matched task-level controls.

ScienceQA-IMG, ChartQA, and COCO each have a 1,000-input panel nested within a larger panel of 1,965, 1,457, and 2,000 independent image clusters, respectively. Every new input is disjoint from the original 48 examples and corresponds to one independent image cluster. We retain continuous template-MV zeroing, seed 42, 32 denoising updates, and 128 response positions, pairing native and intervened generation on the same inputs. Both COCO panels use the same 2,000-image reference collection.

Each model cell gives native → intervened performance and the paired decrease Δ\Delta with its pointwise 95% confidence interval. Intervals use 10,000 paired image-cluster bootstrap resamples, and differences are computed before rounding. QA decreases are in percentage points (pp), and COCO decreases are in CIDEr × 100.

Task and metricInputsLaViDaDream-VLMMaDA
ScienceQA-IMG, accuracy (%)1,00071.40 → 0.00 / Δ 71.40 [68.50, 74.20]92.40 → 0.10 / Δ 92.30 [90.60, 93.90]57.10 → 0.00 / Δ 57.10 [54.10, 60.20]
ScienceQA-IMG, accuracy (%)1,96571.25 → 0.00 / Δ 71.25 [69.26, 73.23]92.01 → 0.05 / Δ 91.96 [90.74, 93.13]56.84 → 0.10 / Δ 56.74 [54.55, 58.98]
ChartQA, relaxed accuracy (%)1,00061.60 → 0.00 / Δ 61.60 [58.50, 64.60]83.30 → 0.10 / Δ 83.20 [80.90, 85.50]10.70 → 0.00 / Δ 10.70 [8.80, 12.60]
ChartQA, relaxed accuracy (%)1,45765.61 → 0.00 / Δ 65.61 [63.21, 68.02]86.07 → 0.07 / Δ 86.00 [84.21, 87.78]8.85 → 0.00 / Δ 8.85 [7.41, 10.30]
COCO, CIDEr × 1001,00064.88 → 0.03 / Δ 64.85 [61.88, 67.85]114.67 → 0.24 / Δ 114.43 [110.02, 119.00]91.16 → 0.21 / Δ 90.95 [87.12, 94.78]
COCO, CIDEr × 1002,00064.50 → 0.03 / Δ 64.47 [62.39, 66.63]116.06 → 0.20 / Δ 115.85 [112.58, 119.08]92.03 → 0.17 / Δ 91.86 [89.09, 94.69]

Every model–task combination shows a decrease at both sample sizes, with all intervals above zero. All nine comparisons within each panel remain significant after Holm correction, with p<0.001p<0.001. The template-MV effect therefore replicates across all three models and tasks on substantially larger independent samples.

The enlarged norm and operation controls directly address what generation needs from the affected representation. On 1,457 ChartQA images, Dream-VL achieves 84.83% when direction is preserved and only the norm is reduced. This tests the behavioral relevance of the scale–direction distinction in Eq. (4). On 2,000 COCO images per model, restoring residual carry (R) outperforms restoring normalized FFN reading (PG) by 55.47, 115.11, and 88.68 CIDEr × 100 for LaViDa, Dream-VL, and MMaDA, respectively, with all paired intervals above zero. These controls share Q/K/V access rules and final-readout protection, testing the functional distinction between reading and carry discussed alongside Eq. (5). The full comparisons appear in Reviewer mGsD W2.

We further test whether an edit confined to one token, one layer, and one update reaches the masked decoder's decisions. On 384 independent COCO images with Dream-VL, we fork the same native state, reduce the MV of the first template token by 25% at the middle layer of the first update, restore the original norm, and then remove the intervention. The mean post-intervention norm ratio is 1.00038. Relative to the native branch, we measure the next block's normalized input, candidate identities across the 128 response positions, and the positions selected for commitment.

ConditionNormalized-input relative L2 change (%)Candidate difference rate (%)Inputs with a changed commitment set (%)
Native0.0000.0000.00
Single MV suppression + norm restoration1.8650.65174.22

The local edit alters the next normalized input and commitment decisions despite norm restoration. Together with the norm and R/PG controls, these measurements test distinct consequences of preserving scale, reading, or carry. This supports specific representation-use predictions with behavioral and computational evidence. W2 adds task-level evidence at matched magnitude, while W4 separately tests final-answer effects under its replay protocol.


AuthorsOpenReview ↗

Response to Reviewer yAdw (2/4)

W2. Repeated interventions, cascading errors, and unmatched random controls.

Response.{\color{green}\mathbf{Response.}} The supplementary window control finds a large coordinate-dependent loss after matching edit magnitude and frequency and fixing targets from the unmodified trajectory. This directly tests whether the amount of repeated perturbation alone can explain the outcome.

For LaViDa on 384 independent ChartQA images, we record the reference MV coordinates and planned perturbation sizes along an unmodified trajectory. Both intervention branches follow this fixed schedule, so later targets are independent of peaks that develop on the perturbed trajectories. Within each window, MV and sparse non-MV edits cover the same template tokens and use the same number of edits. The branches then evolve under these controlled editing conditions, allowing coordinate choice to affect the subsequent trajectory. Comparisons retain pairs meeting the actual L2 matching criterion after BF16 writeback, with maximum relative matching errors below 3% in every reported window. The table reports all eight windows and their qualified sample counts. Scores are accuracy percentages, and differences are sparse-control minus MV accuracy in pp. The layer ranges refer to the declared intervention sites.

Layer × denoising-step windowEdits per target tokenMatched inputs / 384MVEqual-magnitude non-MVDifference [95% CI]
First 1 × first 1138458.5958.590.00 [0.00, 0.00]
First 1 × last 1138457.8158.070.26 [−1.30, 1.82]
Last 1 × first 1138458.5958.590.00 [0.00, 0.00]
Last 1 × last 1138458.5958.590.00 [0.00, 0.00]
First 6 × first 84838458.5958.590.00 [0.00, 0.00]
First 6 × last 8483845.9957.2951.30 [46.09, 56.51]
Last 6 × first 84823458.9758.970.00 [0.00, 0.00]
Last 6 × last 84823757.8158.650.84 [−1.27, 3.38]

The first-six-layer, last-eight-step window separates the two conditions by 51.30 pp [46.09, 56.51], with 5.99% accuracy after MV edits versus 57.29% after matched ordinary-coordinate edits. The early window with the same edit count leaves both branches at 58.59%, and the other windows show no robust additional MV loss. Actual magnitude is matched within each window; total displacement can differ across windows. The decisive late-window contrast shows that editing the reference MV coordinates has a different functional consequence from equally large ordinary-coordinate edits. It establishes a dependence beyond edit size and count under a fixed-reference policy, directly addressing the original unmatched-control concern while locating the effect in a specific layer–step range.


AuthorsOpenReview ↗

Response to Reviewer yAdw (3/4)

W3. Separating empty outputs and premature termination from impaired content processing.

Response.{\color{green}\mathbf{Response.}} The additional diagnostics identify both an output-formation failure and impaired answer discrimination under a valid readout. We first test whether the collapse arises from early computational exit or visible-answer truncation, then examine whether the model can answer correctly once empty outputs and extraction failures are removed from the evaluation.

On 1,000 independent ChartQA image clusters with Dream-VL, we cross native versus template Zero+Norm with native versus controlled stopping. Zero+Norm removes MV and restores the original vector norm. The control suppresses stop tokens at the first three answer positions while those positions remain uncommitted. It is applied to both model states, so the effect of changing the stopping rule is measured for the native model as well. Each input's four branches use the same execution environment.

All four conditions complete 32 forward passes. The empty answers therefore occur despite completing the denoising budget. The table separates correct responses from the different output failures, with every percentage measured over the full panel.

Activation conditionStop-token policyAccuracy (%)Empty (%)Nonempty, unparseable (%)Parseable, incorrect (%)
NativeNative83.400.000.9015.70
Zero+NormNative0.0098.500.700.80
NativeStop tokens blocked at first 3 answer positions68.100.005.5026.40
Zero+NormStop tokens blocked at first 3 answer positions1.3032.0027.9038.80

Stopping control reduces the intervention's empty-output rate from 98.50% to 32.00%, yet accuracy reaches only 1.30%. The control also changes native accuracy, making the comparison under the same stopping rule essential. That comparison leaves a 66.80 pp gap [63.80, 69.90], showing that improved output formation alone does not restore correct answering.

We then test content discrimination with fixed-slot four-choice readout on 256 real ChartQA numerical questions. A complementary panel contains 256 controlled chart pairs, or 512 images. Each pair preserves the question and options while changing the chart values that determine the answer, giving the two images different correct options. Native and intervened branches produce 100% nonempty, parseable outputs in both panels.

Evaluation setEvaluation unitsMetricNative (%)Zero+Norm (%)
Real ChartQA256 independent imagesFour-choice accuracy81.6425.78
Controlled chart pairs512 imagesIndividual-image accuracy100.0024.80
Controlled chart pairs256 pairsPairs with different predicted options100.000.39
Controlled chart pairs256 pairsPairs with both answers correct100.000.00

The real-question accuracy drop is 55.86 pp [48.44, 62.89], estimated with 5,000 paired image-cluster bootstrap resamples. On the controlled charts, the intervention still answers 24.80% of individual images correctly, but changes its option in only 1 of 256 pairs after the decisive values change. No pair has both answers correct. These measurements distinguish generating a valid option from using chart evidence to select the correct one. The degradation therefore extends beyond empty answers or premature truncation to impaired task discrimination, including reduced responsiveness to answer-determining visual changes.


Reviewer_acDXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates massive activations, activations with much larger magnitude than in other tokens, in diffusion multimodal LLMs. Authors investigate the role of these activations and their influence on the model's performance under various interventions.

Critical strengths
  • The paper performs extensive experiments, probing a wide range of properties of massive activations in multimodal diffusion LLMs.
  • Some findings are interesting, for instance how their contribution to QA performance depends on the role of the token they appear in.
Critical weaknesses
  • It is unclear to me what is the motivation and the goal of this paper. Why are we particularly looking at massive activations in multimodal diffusion LLMs?
  • It is unclear what we have practically learned from the experiments. It reads as a series of ablations and their mechanistic evaluation, without an overarching story or motivating factor.
  • The paper is missing a clear discussion on the implications of the multitude of experiments performed in this work.
Questions for authors

Why are we particularly looking at massive activations in multimodal diffusion LLMs? What are we hoping to learn about this particular model class?

What have we practically learned from these experiments, and what are the implications of these findings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer acDX (1/3)

Thank you for your careful reading and for recognizing the breadth of our experiments and their role-dependent effects. We clarify below the central question, the findings that distinguish competing explanations, and their implications for understanding generation.

W1. Motivation and goal of studying massive activations in dMLLMs.

Response.{\color{green}\mathbf{Response.}} Our study asks which computational functions make massive values (MV) important during generation in diffusion multimodal large language models (dMLLMs). Their exceptional magnitude makes them a precise intervention target. The substantive question is what the model needs from the representation containing them: its scale, its reading by normalized downstream operations, or its continued contribution through residual addition. These explanations suggest different interpretations of the damage caused by MV removal.

Prior work provides important starting points. Kovaleva et al. [1] showed sensitivity to outlying LayerNorm dimensions, and Dettmers et al. [2] demonstrated the importance of precision for outlier channels. Most directly, Sun et al. [3] found that replacing massive residual activations with cross-input means largely preserves LLaMA2 performance despite severe damage from zeroing them, supporting a bias-like role. Jin et al. [4] identified context-related effects of large Q/K values, while Roh et al. [5] studied high-magnitude dimensions responsive to domain-specific patterns. Gan et al. [6] and Turri et al. [7] connected massive activations in image diffusion Transformers to local detail synthesis and semantic changes. These precedents motivate testing which aspects of the affected representation actually support computation in dMLLMs.

Multimodal masked generation adds a specific reason to investigate this dependence. The tested decoders predict multiple unresolved response positions and select which positions to commit. An activation change can therefore affect both the predicted content and when that content enters the context for subsequent updates. The evolving response is also conditioned on the image and question, making template- and response-region representations relevant to how multimodal evidence is used throughout generation.

Our contribution is to separate these functional dependencies and connect them to the decoder's actual decisions. The operation controls establish that restoring normalized FFN reading does not substitute for restoring residual carry in the tested template-captioning settings. A separate local experiment directly observes changes in masked commitment after a norm-restored MV edit. Together, these results connect the functional use of MV-containing representations with the additional decisions required by the tested masked-update rule.

[1] Kovaleva et al. BERT Busters: Outlier Dimensions that Disrupt Transformers.
[2] Dettmers et al. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.
[3] Sun et al. Massive Activations in Large Language Models.
[4] Jin et al. Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding.
[5] Roh et al. Embracing Anisotropy: Turning Massive Activations into Interpretable Control Knobs for Large Language Models.
[6] Gan et al. Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers.
[7] Turri et al. Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers.


AuthorsOpenReview ↗

Response to Reviewer acDX (2/3)

W2. The experiments need an overarching question and a clear progression.

Response.{\color{green}\mathbf{Response.}} The experiments progressively distinguish where generation depends on MV, which representation properties matter, and which operations sustain that dependence. Applying the same intervention rule to image, question, template, and response regions first identifies sensitive representations. Complementary norm controls then distinguish scale contraction from changes in the representation's composition. On 1,457 ChartQA examples with Dream-VL, removing template MV and restoring the norm yields 0.07% accuracy, whereas preserving direction while reducing the norm yields 84.83%. Thus, scale contraction alone cannot explain this template-region failure. The subsequent peak-only and amplification controls further limit explanations based solely on the peak's presence or magnitude.

The operation experiments address the stronger question left open by these results: how does the affected representation contribute downstream? After separating Q/K and V access, we distinguish its additive residual contribution from its reading by the normalized FFN. R restores residual carry while retaining the perturbed FFN input, while PG restores normalized FFN reading and retains the perturbed carry. Both conditions use the same Q/K/V access rules and terminal-readout protection. Each branch selectively restores an operation while its subsequent hidden states evolve under that intervention.

On 2,000 COCO images, R versus PG yields 55.50 versus 0.03 for LaViDa, 115.11 versus 0.00 for Dream-VL, and 88.72 versus 0.03 for MMaDA (CIDEr × 100). All paired confidence intervals for the differences exclude zero. This is the central functional finding: repairing the normalized FFN's reading of the representation leaves captioning severely impaired, while restoring its residual continuation preserves substantially more performance. The progression therefore leads from regional sensitivity to a concrete operation-level distinction. It narrows the explanation beyond “large values matter” or “removal changes the norm” and identifies the residual contribution as an important part of the tested dependence.


W3. Implications of the experimental findings.

Response.{\color{green}\mathbf{Response.}} The findings shift the interpretation of MV from numerical prominence to the computational role of the representation containing it. In the tested template settings, preserving vector length, preserving a peak, and restoring a normalized reader are different from preserving the representation's continued residual contribution. Template-region sensitivity in visual tasks also shows why a token's nominal role does not determine its functional importance: a template representation can support the computation needed to answer an image-dependent question. The carry/reading contrast makes that observation more specific by identifying operations with sharply different recovery outcomes.

This distinction has already improved our diagnosis of generation failures. In Dream-VL's ChartQA experiments, a matched stopping control improves output formation but leaves a large accuracy gap. We then impose fixed-slot four-choice readout on 256 real ChartQA questions: both native and intervened branches produce 100% nonempty, parseable outputs, yet accuracy falls from 81.64% to 25.78% under template MV removal with norm restoration. The decline is 55.86 percentage points, with a paired 95% confidence interval of [48.44, 62.89]. Thus, restoring a usable output format does not restore task discrimination. This is a demonstrated diagnostic consequence: the observed failure extends beyond empty answers or premature termination.


AuthorsOpenReview ↗

Response to Reviewer acDX (3/3)

Q1. Why study MV in dMLLMs, and what do we hope to learn about this model class?

Response.{\color{green}\mathbf{Response.}} Masked generation adds a commitment decision to the question of how MV affect prediction. Under confidence-based selection, changing confidence can change when a position is committed even if its preferred token remains unchanged. Because committed content becomes context for later updates, this decision is a concrete point at which an internal representation can influence the evolving response. We therefore examine both the operations using that representation and its effect on the masked update.

The local Dream-VL experiment directly connects this motivation to observation. On 384 COCO images, we suppress the first template token's MV by 25% at one layer of the first update, restore its norm, and remove the intervention. The mean norm ratio is 1.00038, yet the next normalized input changes by 1.865% in relative L2 and 74.22% of inputs have a different commitment set. Scale preservation therefore leaves a measurable effect on the masked decoder's current decision. A separate replay experiment uses a single 25% MV suppression without norm restoration, crosses native and perturbed candidate/position decisions, and finds no robust final-score degradation. Together, the experiments show how to distinguish a change in masked decisions from evidence that such a change affects final performance.


Q2. What have we practically learned, and what are the implications?

Response.{\color{green}\mathbf{Response.}} We have learned which distinctions are necessary to interpret an MV intervention. In the three models' tested COCO template settings, restoring normalized FFN reading does not replace restoring residual carry. The recovery comparison therefore directs failure analysis toward the representation's continued residual contribution, beyond checking what the FFN reads. The local measurements and the separate replay protocol add a complementary lesson about distinguishing current-update sensitivity from final-answer impact. These findings give concrete computational targets and outcome measures for tracing the consequences of an intervention.

The valid-readout experiment demonstrates why this matters in practice: making every answer parseable still leaves substantial loss of correct answering. Failure analysis can therefore distinguish output formation from task discrimination, then use selective operation restoration to investigate the affected computation. Future quantization, cross-step reuse, or hallucination interventions can use these distinctions to formulate testable designs; their efficiency and quality benefits remain prospective. The present contribution is the experimentally supported functional decomposition and the diagnosis it enables.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.