- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes to change how masked diffusion models (MDMs) are trained: instead of using random masking patterns at training, the authors propose to estimate token-wise importance scores (where tokens with higher importance are e.g. those that carry crucial semantic meaning), and increasing the probability of retaining high-importance tokens at high-noise levels, while reducing the probability of retaining them at low-noise levels. This is supposed to provide important context at high noise, while improving the model's capability of predicting important tokens at low noise. The authors also propose changing the MDM sampler at inference, to take token importance into account alongside token-wise confidence scores.
Critical strengths
The idea of treating tokens differently based on some notion of token importance for MDM training is interesting, and I believe worthwhile exploring; clearly, there should be some headroom over uniform random masking, as is commonly done. So the idea of the paper is interesting. The authors also show it beats simple uniform unmasking at inference.
Critical weaknesses
There are several weaknesses (from most to least severe):
- the training objective is defined in inconsistently. Sometimes the factor 1/t is said to be explicitly used, while at other times it is omitted (e.g. equation 11, algorithm 1). Without it, switching off importance weighting and the masking curriculum does not recover the MDLM NELBO, contrary to the claim below eq. 12. It's unclear whether the factor was used in the experiments.
- the sampler is defined inconsistently: in section 3.4, high-importance tokens are committed first, while algorithm 2 selects based on lowest entropy. Also, the sampler's effectiveness remains unclear from the experiments: the obvious experiment to run would be a compute-matched comparison with the proposed sampler and one or two standard MDM samplers (e.g. random, confidence based), but this is missing.
- importance score computation/estimation is questionable: the importance scores are trained on representations derived from masked sequences; however, it is then applied on clean sequences to decide which tokens to mask in the next step. This inconsistency is not addressed, and it remains open why this design was chosen, and whether a head trained on clean sequences would work better. Then again at inference, the inputs come from corrupted sequences, so two separately trained prediction heads seem like a more reasonable approach, or at least this should be addressed by the authors. Also, it remains open whether prediction difficulty really correlates with semantic importance (the ablation showing correlation with the teacher's surprise targets doesn't answer this, because that's just comparing with the targets used in training the prediction head).
- Lemma 7 doesn't apply to the losses used in the paper: it assumes a token's loss only depends on whether that position is masked or not, while in reality, it depends on the entire (partially masked) context. This means Theorem 6 is not established. Corollary 9 suffers from a similar mistake. It also asserts cross-entropy is bounded by log K, which is incorrect.
- the experimental results are insufficient in supporting their training algorithm. While they report that it helps (in terms of PPL) over vanilla uniform masking, the authors don't compare against existing methods from the literature that try to solve the same problem, such as [1], [2]
- large parts of the paper read extremely AI-generated, to the point that sentences are often entirely incomprehensible to me (examples: "which sequence the estimator reads is load-bearing, and importance has two uses with different admissibility requirements", "Because the design in Table 3 (b) is complete, its marginal effects are interpretable", "effects near that scale we mark unresolved", etc.) or superfluous ("Property (iii) is conditional, and we report the condition rather than assume it")
- some reported results are wrong / inaccurate. E.g.:
- Figure 1: says in the figure that at high noise, important tokens are preserved (should be the opposite)
- Section 4.1 reports 9B training tokens, but Appendix C says 37M * 1024 = 37B tokens
- Table 12: should report 67.01 instead of 27.04 (see [3])
[1] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and Generalized Masked Diffusion for Discrete Data. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. arXiv:2406.04329.
[2] Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Anchored Diffusion Language Model. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 2025. arXiv:2505.18456.
[3] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and Effective Masked Diffusion Language Models. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. arXiv:2406.07524.
Questions for authors
My recommendation to the authors would be to revise the paper and fix the above issues, and then resubmit a reworked version.