acceptodds
Under review as a conference paper at ICLR 2027

Layer Collapse in Diffusion Language Models

Abstract

Diffusion language models (DLMs) have recently emerged as competitive alternatives to autoregressive (AR) language models, yet the differences in their activation dynamics remain poorly understood. We characterise these dynamics in LLaDA-8B and identify a striking layer-collapse property: early layers exhibit highly similar, collapsed activation patterns, and a single large super-outlier channel dominates the activations at every token position in layers 1–24 (of 32). Despite its apparent redundancy, we have found that this outlier is critical: pruning it causes the model's outputs to degrade into repetitive random token loops. Paradoxically, besides this outlier, layers in LLaDA contain more redundant representations, where redundancy is particularly pronounced in earlier layers. This pattern is the reverse of what is commonly observed in AR language models, where deeper layers tend to become more redundant due to undertraining, as measured by representation similarity. Our analysis further indicates that layer collapse in DLMs is not driven by undertraining. Rather, it appears to arise from overtraining: a dominant outlier becomes an indispensable carrier of information, while the remaining representations collapse into redundant structure. These observations have strong practical implications, as we verify through controlled pre-training experiments. First, DLMs are surprisingly robust to compression: performance of LLaDA under 3-bit GPTQ quantization drops by only 1.8% on GSM8K, whereas Llama-3.1-8B under the same settings drops by 64.7%. Second, optimal non-uniform sparsity allocation reverses between the two model families: under an average budget of 50% sparsity, allocating more sparsity to early layers in LLaDA yields +8.4% over the reverse strategy, while for Llama the same early-layer-sparse allocation incurs -8.4%. Our findings reveal that the DLM training objective fundamentally reshapes layer dynamics relative to AR models, with direct consequences for how such models should be compressed and deployed. Our code will be made available upon publication.

Primary area:
learning theory
Keywords:
Deep learning, Diffusion, Curse of depth, learning dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ccTDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Masked diffusion language models (DLMs) offer alternative generation dynamics to standard autoregressive models. While prior work has already characterized the difference in attention behavior (Myrzakhan et al., 2026), the present work analyzes the activation structure in DLMs.

In line with results reported in an unpublished contemporary work (Goel et al., 2026), this work finds that early DLM layers feature high redundancy evidenced by high activation cosine similarity between consecutive layers. Interestingly, the authors find that both LLaDA-8B and iLLaDA-8B feature very large super-outlier channels in the first 1--24 layers, which is different comparing to a Llama-3.1-8B featuring the same underlying architecture. A weight spectrum analysis attributes this outliers to overfitting. This insight is then incorporated into pruning of Llada-8B and Llama-3.1-8B, where Llada-8B shows higher robustness to pruning and quantization. In order to generalize the findings, a Pythia-160M architecture is trained from scratch with both masked and AR training loss, finding that the same redundancy behavior is present in this small scale too.

Critical strengths

The super outliers in the the early layers of Llada-8B and iLlada-8B are very interesting and may spark additional research in pretraining approaches.

Critical weaknesses

My main reservations are regarding overclaiming/overgeneralizing certain insights, which are not supported by empirical evidence. For example, while the authors attribute super outliers to the diffusion training, it is only found in one particular architecture of the same size (Llada and iLlada share large parts of the architectural details). Indeed, one may consider different from-scratch trained models, such as Llada MoE v2 or even looking at well established models in the vision domain. Moreover, the findings do not clearly translate to actionable methods: The derived early-is-sparser pruning does not significantly work better than uniform or deep-is-sparser pruning technique. The current quantization experiment does not even consider mixed-precision quantization.

Questions for authors
  • Adding more models to the investigation would strengthen the claim that the training is the root cause of super outliers, not the architecture.
  • I would consider mixed-precision quantization, e.g., by quantizing early layers to 2 bits and later ones to 3 bits.
  • It would be interesting to put the pruning and quantization in context to prior work (Myrzakhan et al., 2026; Goel et al., 2026). What is most effective? Pruning token positions, removing entire layers, quantizing, etc.?
  • How do the super outliers behave across the sequence dimension? Are all positions equally emitting the outliers?
  • Line 461: "... removing it causes total model collapse, a fragility with no parallel in AR models of comparable scale." We only have evidence on Llama 3.1, no generalization to other AR models is shown.
Reviewer_ppq3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares the internal representations of diffusion language models (DLMs) and autoregressive (AR) models, identifying a persistent outlier across multiple layers and high representational similarity among early layers in LLaDA-8B and iLLaDA-8B. The authors associate these phenomena with overtraining of early layers. They also claim that DLMs are more robust to compression than AR models and benefit from more aggressive pruning of early layers.

Critical strengths
  • The paper shows that a single activation channel dominates across multiple layers in LLaDA-8B and iLLaDA-8B, and removing it causes model collapse. This observation is interesting.
Critical weaknesses
  1. The generalizability of the findings is highly limited. The main analyses rely on LLaDA-8B, iLLaDA-8B, and a single Llama model. The outlier is observed only in two closely related models, making it difficult to draw general conclusions about differences between DLMs and AR models. Furthermore, the 160M comparison is insufficient to isolate the diffusion objective's effects.

  2. The mechanistic explanation and supporting compression experiments are insufficient. The high cosine similarity is dominated by the outlier and does not necessarily indicate layer redundancy. The Alpha-Hill differences are also limited, and iLLaDA does not consistently reproduce the same trend, providing insufficient evidence for the overtraining interpretation. Compression results depend on the models, tasks, and methods used, and low-bit quantization experiments are limited to GPTQ. Moreover, the paper establishes no clear connection between the outlier observation and compression robustness.

Questions for authors
  • Is there evidence that similar outliers occur in DLMs outside the LLaDA family? In other words, how generalizable are the paper's findings?

  • Is there experimental evidence that the outlier and layer redundancy patterns claimed to be specific to DLMs directly contribute to compression robustness? Do similar trends hold with other compression methods specifically designed for low-bit quantization?

Reviewer_YkbJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper compares the activation dynamics of two diffusion LMs trained from scratch, LLaDA-8B and iLLaDA-8B, with those of the autoregressive Llama-3.1-8B. It reports a single channel (3848 in LLaDA, 2658 in iLLaDA) that is large at every token position in layers 1–24. Pruning this channel collapses GSM8K accuracy (Table 1). The paper also reports an "inverted" cross-layer cosine-similarity structure: early LLaDA layers are highly redundant, partly because of this channel (Fig. 3). Using the Hill estimator of weight-spectrum tails (αHill), the authors attribute this early-layer redundancy to relative overtraining rather than undertraining (Sec. 2.4, Fig. 4). Under 3-bit GPTQ and 50% WANDA pruning, LLaDA degrades far less than Llama-3.1-8B. LLaDA also favors allocating more sparsity to earlier layers, the reverse of Llama (Sec. 3, Fig. 5). Finally, the authors pre-train a 160M-parameter AR/masked-diffusion pair on the same 100B FineWebEdu tokens. From this pair they argue that the similarity pattern, the spectral signature, and the compression behavior all stem from the diffusion objective (Sec. 4, Table 2, Figs. 6–7).

Critical strengths
  1. The paper identifies a concrete phenomenon in from-scratch DLMs and tests it causally. In LLaDA-8B, channel 3848 is the largest plotted channel in layers 1–24. In layers 1–23 it exceeds each of the other four plotted channels by at least 2.1× (Fig. 2a, p. 4). Its "max" and "mean" curves in Fig. 1 (p. 2) nearly coincide (e.g., ≈23.5 vs. ≈22.0 at L15), unlike Llama's channels. This is consistent with a position-independent channel. Its magnitude barely changes across diffusion steps (App. B.3, Fig. 11), and the pattern persists without masking (App. B.1, Figs. 8–9). Pruning it lowers GSM8K accuracy from 83% to 0% for LLaDA, and pruning channel 2658 lowers iLLaDA from 78% to 0.5%; the outputs degenerate into repetitions (Table 1 and example, L. 175–191). The DREAM-7B/Qwen-2.5-7B comparison (Fig. 3) is a useful control: DLMs initialized from AR weights appear to keep AR-like layer dynamics. This matters because DLM compression and interpretability methods would need to handle such a channel explicitly.
  2. The compression consequences are large and practically relevant. From the plotted values in Fig. 5 (p. 7), 3-bit GPTQ lowers LLaDA's GSM8K accuracy from 64.7% to 62.9%, whereas Llama-3.1-8B falls from 73.0% to 8.3%. At 50% uniform WANDA sparsity, LLaDA keeps 64.0% on GSM8K versus 32.8% for Llama (the "almost double" of L. 311–312). On QnA the direction holds after normalizing for chance: at 3 bits, LLaDA retains 94% of its above-chance accuracy and Llama 66%. The matched 160M pair is a welcome attempt at a control. In it, pruning robustness also holds after normalizing for chance (92% vs. 72% retained at 50% uniform sparsity; Fig. 7). The authors also report results that cut against their account: iLLaDA's weak αHill dip (App. B.4), the absence of a super-outlier at 160M (L. 367), and AR-160M leading at 3–4 bits (L. 398). If these consequences generalize, AR-derived compression heuristics should not be applied to DLMs unchanged.
Critical weaknesses
  1. The "overtraining" explanation is asserted rather than identified, and the paper's own models dissociate the spectral signature from the super-outlier. An "overtrained" layer is defined only as one with relatively low αHill (L. 258–260). The paper says early-layer redundancy "arises from the relative overtraining of these layers" (L. 064–065) and that Fig. 4 "shows that these layers are actually overtrained" (L. 278–279). Both statements therefore re-describe the spectral measurement rather than explain it. No evidence links the dip to training duration, although the authors trained the 160M models themselves and could track αHill and similarity across checkpoints or token budgets. More importantly, the dip and the outlier do not co-occur in the paper's own models. (i) In LLaDA (reading Fig. 4, p. 6), αHill is below Llama's minimum (≈3.49) only in layers 0–3 (≈2.74–3.42) and reaches Llama's level by layer 6 (≈4.04). The super-outlier, however, persists through layer 24, and the redundant block spans "15+ layers" (L. 211). (ii) iLLaDA has the outlier (Table 1) and early-layer redundancy (Fig. 3), yet its dip is confined to layers 0–1 (App. B.4, L. 888–890). (iii) DLM-160M shows the dip (Fig. 6) but no super-outlier (L. 367). By the paper's own reading (L. 259–260), LLaDA's last eight layers would count as relatively undertrained (mean αHill ≈4.91 vs. ≈4.26 for Llama), so the spectra show a steeper depth gradient rather than overtraining as such. "Overtraining" also cannot mean "more training" in absolute terms. LLaDA-8B was pre-trained on about 2.3T tokens, versus about 15T for Llama 3 8B, as listed in the LLaDA report (Nie et al.). In the 160M pair, both models see the same 100B tokens, and the DLM's loss covers only masked positions (App. A.2(iii), L. 745–746). αHill is also averaged over modules (Fig. 4 caption, L. 279) whose shapes differ between the two 8B models. Per the LLaDA report, LLaDA uses 32 key/value heads and an FFN width of 12,288, versus 8 and 14,336 for Llama 3 8B. Finally, the HT-SR pruning method the paper cites (AlphaPruning; Lu et al., 2024) treats low-αHill layers as better trained and prunes them less, the opposite of the paper's prescription. This matters because overtraining is the paper's main explanatory claim (abstract, L. 020–023; Discussion, L. 461–463). At present the spectra are at most consistent with it.
  2. Neither the attribution of the observed behaviors to the diffusion objective nor their generalization to "DLMs" is established. At 8B, the evidence compares one AR model with two from-scratch DLMs from the same line of work (Nie et al., 2026a;b), and only LLaDA appears in the compression experiments (Sec. 3). Besides the objective, LLaDA and Llama-3.1-8B differ in attention masking, attention/FFN shapes, tokenizer, data, token budget (≈2.3T vs. ≈15T, i.e., ≈6.5×), training recipe, and the instruction tuning of the GSM8K variants. Degradation from post-training quantization is known to grow with the amount of pre-training data (Kumar et al., 2025). Llama's 3-bit GSM8K collapse (from ≈963 to ≈109 of 1,319 problems, if Fig. 5 uses the full test set) may therefore partly reflect its token budget rather than its objective. An AR model with a comparable budget (e.g., Llama-2-7B, 2T tokens) is the missing control. The paper's own Fig. 3 shows that a DLM fine-tuned from AR weights (DREAM-7B) lacks the pattern, so the claims concern at most from-scratch DLMs. The 160M pair was designed to isolate the objective, but it cannot carry the attribution: (i) it lacks the super-outlier (L. 367), so it cannot test the central mechanism; (ii) it changes the attention mask as well as the loss (App. A.2(i), L. 744–745), despite "solely varying in the training objective" (L. 357); (iii) it has one run per learning rate (L. 741); and (iv) its compression results are near chance and mixed (Weakness 3). The paper states that "the diffusion objective is responsible for the observed layer and compression behaviours" (L. 117–118) and that "we could thus ensure that these observations are due to the DLM training objective" (L. 365–366). Both statements go beyond the evidence. This matters because attribution to the objective is what turns two model-specific observations into general guidance for "DLMs" (abstract, L. 030–032; contribution 3, L. 114–116).
Questions for authors
  1. What is the operational definition of "overtraining", and can it be tested directly? For example, do the early-layer αHill dip and the cross-layer similarity of DLM-160M grow across training checkpoints or with the token budget? Which modules drive LLaDA's dip, and do they write into or read from channel 3848? How do you reconcile iLLaDA (outlier and redundancy, but a dip only in layers 0–1) and DLM-160M (dip but no outlier) with the proposed causal chain?
  2. Suppose an AR model with a token budget comparable to LLaDA's (e.g., Llama-2-7B, 2T tokens) is compressed with identical GPTQ/WANDA settings. Does it show the same 3-bit GSM8K collapse and the same EIS/DIS preference as Llama-3.1-8B? If instead it behaves like LLaDA, the robustness would be better explained by training tokens than by the objective.
  3. Does replacing the super-outlier by its per-layer mean, rather than by zero, preserve accuracy? For a like-for-like AR comparison, what happens when Llama-3.1-8B's massive-activation dimensions or super weights are ablated? Please also report a random-channel control and the effect of ablating iLLaDA's second channel (≈3512), and specify which tensor and layers the ablation modifies.
  4. Can you report OWL and AlphaPruning allocations on both models, several ϵ values, and variability over calibration draws, with paired tests on the same GSM8K items? Do the QnA differences of ≈0.5 pp at 30–50% sparsity survive?
  5. For the 160M pair, can you report multiple seeds per configuration? Which learning rate's model underlies Table 2, Fig. 6, and the quantization panels? What is the robustness after normalizing for chance? Is there an intermediate scale (e.g., ≈1B parameters) at which the controlled DLM develops a super-outlier while its AR twin does not?

Clarification requests

  1. Which tensor is used in Figs. 1–3, in Eq. 1, and in the Table 1 ablation: the residual stream after each block, or the post-RMSNorm input to the QKV projections (as in Fig. 2)? Are the first position and the BOS token included? Why do Llama's massive activations not appear in Fig. 1?
  2. How are the multiple-choice QnA tasks scored for LLaDA and DLM-160M: likelihood-based ranking or generation? Why do Table 1's GSM8K baselines (83% for both models) differ from the Fig. 5 values (≈64.7% and ≈73.0%)? How was the 200-question subset chosen, and does Fig. 5 use the full 1,319-problem test set?
  3. Which GPTQ settings were used (group size, act-order, symmetric vs. asymmetric, embeddings and LM head), and what calibration sequence length? Were the settings identical for both models? Does the EIS/DIS schedule apply to every linear layer of a block?

Minor

  • L. 025–030, 053; Table 1 (L. 183–191): the abstract's "1.8%", "64.7%", "+8.4%", and "-8.4%", and Table 1's "Δ baseline", are absolute percentage-point differences; please write "pp". In relative terms, Fig. 5 implies 3-bit GSM8K drops of 2.7% (LLaDA) and 88.7% (Llama). The plotted values (≈853 → 830 of 1,319 problems) give 1.74 pp for LLaDA; "1.8" presumably comes from rounding both accuracies first. Also, "-8.4%" (L. 030) uses a hyphen instead of a minus sign.
  • L. 014, 049, 100, 203, 460 state "layers 1–24 (of 32)". However, Fig. 1's L25 row (L. 094–097) still marks ch 3848 as the largest channel (≈17.0 vs. ≈16.7 for ch 753). In Fig. 2a it also remains, barely, the largest plotted channel at layer 25. Please state the layer-indexing convention (block input vs. output; Fig. 3 starts at layer 0) and the criterion for "persists".
  • Fig. 1 caption (L. 102–103): "For Llama, the dominant channel changes on almost every layer." In Fig. 10, however, ch 2932 is the top channel in 5 of the 11 sampled layers (L9, L18, L24, L27, L30), and ch 291 in 3 (L12, L15, L21). The row labels ("L1 Act") and the "max"/"mean" statistics are also not defined (taken over positions, over samples?).
  • L. 329–330: "although Llama is ahead again at 4 bits, and at both 3 and 4 bits for GSM8K" can be read as saying that Llama leads on GSM8K. Fig. 5 shows LLaDA ahead at both 3 bits (≈62.9% vs. ≈8.3%) and 4 bits (≈61.9% vs. ≈50.3%). Please rephrase.
  • Terminology. "Significant overfitting" (L. 113) is used alongside "overtraining" elsewhere (e.g., L. 021, 065, 259–260, 462). Overfitting implies a generalization gap, which is not measured. "Collapse" refers both to representational similarity ("layer collapse") and to capability loss ("near-total collapse in capability", L. 051; "total model collapse", L. 460). "DL model" (L. 107, 117) should be "DLM". Llama-3.1-8B is called LLaDA's "twin-model" (L. 263) despite the differences listed in Weakness 2.
Reviewer_Zv4GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work paper studies the differences in activation dynamics between masked DLMs (diffusion language models) and autoregressive (AR) models. The authors find that masked diffusion models have high similarity between their early layers representations as well as a dominant activation channel: a 'super-outlier'. They also report better robustness to compression and different pruning strategies.

Critical strengths

Understanding DLMs internal representations and how they differ from those of AR models is an important problem and work on this topic is valuable. The authors study several models, complement their work with compression experiments and a controlled pretraining setting.

Critical weaknesses
  • The observations are interesting enough to justify a more in depth study, but in its current form I find the claims and interpretations lacking sufficient experimental evidence. A more rigorous analysis of each phenomenon would elevate these from simple observations to insights (see the question below on super outlier for an example of this). High representation similarity also does not imply functional redundancy.
  • The discrete diffusion space is advancing fast, and mask diffusion is not representative of all diffusion processes. While the authors make general statements about DLMs in general, they in fact only look at mask diffusion, a particular form of discrete diffusion. I think that distinction should be made clearer and the scope of the statements should be restricted to what they apply to. With respect to the generality of the claims, the complete narrative seems to be mostly supported by LLaDA, the early layer spectral signature used to argue for overtraining seems more absent in LLaDA and the 160M model does not reproduce the dominant channel.

As a side note, while I am familiar enough with the subject to follow the narrative, I think the writing lacks clarity and would probably confuse the average reader. Diffusion language models are not properly introduced, training and the DLM objective are discussed in prose but never formally stated.

Questions for authors
  1. Other types of noises could be discussed. For instance, results on uniform diffusion could support the generality of the findings, or provide yet another angle.
  2. What does the 'super-outlier' actually do? Insights into its computational function, what information it carries, etc., would be valuable. Degradation from ablating it is not particularly informative on its own given its large magnitude. A constant bias that the model relies on would also cause a big disruption if removed, while carrying no input-specific information at all. I thus believe that more in-depth analysis on the outlier's role is warranted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.