- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Masked diffusion language models (DLMs) offer alternative generation dynamics to standard autoregressive models. While prior work has already characterized the difference in attention behavior (Myrzakhan et al., 2026), the present work analyzes the activation structure in DLMs.
In line with results reported in an unpublished contemporary work (Goel et al., 2026), this work finds that early DLM layers feature high redundancy evidenced by high activation cosine similarity between consecutive layers. Interestingly, the authors find that both LLaDA-8B and iLLaDA-8B feature very large super-outlier channels in the first 1--24 layers, which is different comparing to a Llama-3.1-8B featuring the same underlying architecture. A weight spectrum analysis attributes this outliers to overfitting. This insight is then incorporated into pruning of Llada-8B and Llama-3.1-8B, where Llada-8B shows higher robustness to pruning and quantization. In order to generalize the findings, a Pythia-160M architecture is trained from scratch with both masked and AR training loss, finding that the same redundancy behavior is present in this small scale too.
Critical strengths
The super outliers in the the early layers of Llada-8B and iLlada-8B are very interesting and may spark additional research in pretraining approaches.
Critical weaknesses
My main reservations are regarding overclaiming/overgeneralizing certain insights, which are not supported by empirical evidence. For example, while the authors attribute super outliers to the diffusion training, it is only found in one particular architecture of the same size (Llada and iLlada share large parts of the architectural details). Indeed, one may consider different from-scratch trained models, such as Llada MoE v2 or even looking at well established models in the vision domain. Moreover, the findings do not clearly translate to actionable methods: The derived early-is-sparser pruning does not significantly work better than uniform or deep-is-sparser pruning technique. The current quantization experiment does not even consider mixed-precision quantization.
Questions for authors
- Adding more models to the investigation would strengthen the claim that the training is the root cause of super outliers, not the architecture.
- I would consider mixed-precision quantization, e.g., by quantizing early layers to 2 bits and later ones to 3 bits.
- It would be interesting to put the pruning and quantization in context to prior work (Myrzakhan et al., 2026; Goel et al., 2026). What is most effective? Pruning token positions, removing entire layers, quantizing, etc.?
- How do the super outliers behave across the sequence dimension? Are all positions equally emitting the outliers?
- Line 461: "... removing it causes total model collapse, a fragility with no parallel in AR models of comparable scale." We only have evidence on Llama 3.1, no generalization to other AR models is shown.