acceptodds
Withdrawn from ICLR 2027

Not All Tokens Are Equal: Importance-Aware Masking for Discrete Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Masked discrete diffusion language models (DLMs) corrupt every position with the same probability, so a content word carrying eight bits of surprisal and a function word carrying two are treated identically by the forward process. We introduce ATAT, which makes the corruption depend on which token occupies a position and on when it is corrupted: a continuous importance score preserves informative tokens as context at high noise and concentrates training signal on them at low noise. Two choices make this work. The curriculum is applied as a per-token time warp of the survival probability, expressing the masking probability exponentially through a time-warping exponent on the base survival schedule, rather than as the multiplicative rescaling used by prior non-uniform schemes; for the log-linear schedule and small curriculum bounds we prove this keeps the process a valid absorbing-state chain reaching the fully masked state generation starts from, and we measure rather than assume its masking budget, which departs from the uniform rate by at most 0.014. And the signal comes from the model itself: a masked diffusion model already predicts, at every masked position, a distribution estimating how surprising the underlying token is, so an exponential moving average of the denoiser supplies both representation and supervision, and no autoregressive teacher is needed for supervision, initialisation or inference. In a controlled study holding architecture, data order, optimiser and token budget fixed, importance-aware masking reduces perplexity by 7.4%. That headline does not survive a compute correction: ATAT costs 2.18 times more per training token, and a uniform baseline given the same wall clock leaves 5.2%. We report the contrasts behind it rather than defend it whole: the conditioning input is worth 0.3-0.4 PPL, and placing the curriculum in the exponent rather than multiplicatively is worth 1.0 PPL at matched cost and masking budget, the one comparison the compute correction does not touch.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Masked Diffusion Language Models, Importance-Aware Masking, Discrete Diffusion, Dynamic Noise Schedule, Curriculum Learning, Token Surprisal, Efficient Training

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_n7F3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to change how masked diffusion models (MDMs) are trained: instead of using random masking patterns at training, the authors propose to estimate token-wise importance scores (where tokens with higher importance are e.g. those that carry crucial semantic meaning), and increasing the probability of retaining high-importance tokens at high-noise levels, while reducing the probability of retaining them at low-noise levels. This is supposed to provide important context at high noise, while improving the model's capability of predicting important tokens at low noise. The authors also propose changing the MDM sampler at inference, to take token importance into account alongside token-wise confidence scores.

Critical strengths

The idea of treating tokens differently based on some notion of token importance for MDM training is interesting, and I believe worthwhile exploring; clearly, there should be some headroom over uniform random masking, as is commonly done. So the idea of the paper is interesting. The authors also show it beats simple uniform unmasking at inference.

Critical weaknesses

There are several weaknesses (from most to least severe):

  • the training objective is defined in inconsistently. Sometimes the factor 1/t is said to be explicitly used, while at other times it is omitted (e.g. equation 11, algorithm 1). Without it, switching off importance weighting and the masking curriculum does not recover the MDLM NELBO, contrary to the claim below eq. 12. It's unclear whether the factor was used in the experiments.
  • the sampler is defined inconsistently: in section 3.4, high-importance tokens are committed first, while algorithm 2 selects based on lowest entropy. Also, the sampler's effectiveness remains unclear from the experiments: the obvious experiment to run would be a compute-matched comparison with the proposed sampler and one or two standard MDM samplers (e.g. random, confidence based), but this is missing.
  • importance score computation/estimation is questionable: the importance scores are trained on representations derived from masked sequences; however, it is then applied on clean sequences to decide which tokens to mask in the next step. This inconsistency is not addressed, and it remains open why this design was chosen, and whether a head trained on clean sequences would work better. Then again at inference, the inputs come from corrupted sequences, so two separately trained prediction heads seem like a more reasonable approach, or at least this should be addressed by the authors. Also, it remains open whether prediction difficulty really correlates with semantic importance (the ablation showing correlation with the teacher's surprise targets doesn't answer this, because that's just comparing with the targets used in training the prediction head).
  • Lemma 7 doesn't apply to the losses used in the paper: it assumes a token's loss only depends on whether that position is masked or not, while in reality, it depends on the entire (partially masked) context. This means Theorem 6 is not established. Corollary 9 suffers from a similar mistake. It also asserts cross-entropy is bounded by log K, which is incorrect.
  • the experimental results are insufficient in supporting their training algorithm. While they report that it helps (in terms of PPL) over vanilla uniform masking, the authors don't compare against existing methods from the literature that try to solve the same problem, such as [1], [2]
  • large parts of the paper read extremely AI-generated, to the point that sentences are often entirely incomprehensible to me (examples: "which sequence the estimator reads is load-bearing, and importance has two uses with different admissibility requirements", "Because the design in Table 3 (b) is complete, its marginal effects are interpretable", "effects near that scale we mark unresolved", etc.) or superfluous ("Property (iii) is conditional, and we report the condition rather than assume it")
  • some reported results are wrong / inaccurate. E.g.:
    • Figure 1: says in the figure that at high noise, important tokens are preserved (should be the opposite)
    • Section 4.1 reports 9B training tokens, but Appendix C says 37M * 1024 = 37B tokens
    • Table 12: should report 67.01 instead of 27.04 (see [3])

[1] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and Generalized Masked Diffusion for Discrete Data. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. arXiv:2406.04329.

[2] Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Anchored Diffusion Language Model. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 2025. arXiv:2505.18456.

[3] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and Effective Masked Diffusion Language Models. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. arXiv:2406.07524.

Questions for authors

My recommendation to the authors would be to revise the paper and fix the above issues, and then resubmit a reworked version.

Reviewer_qsN3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ATAT, an importance-aware masking framework for discrete diffusion language models. ATAT estimates token importance by combining contextual surprisal obtained from an EMA self-teacher with corpus-level token frequency, without relying on an external autoregressive teacher. It further introduces a time-warped masking schedule designed to preferentially mask important tokens at low noise levels while preserving them as contextual information at high noise levels.

Critical strengths

S1. The paper addresses the limitations of uniform masking in discrete diffusion language models and proposes an intuitive importance-aware curriculum that adapts to the noise level.

S2. The paper provides controlled experiments examining the contributions of importance estimation, masking schedules, and loss weighting.

S3. The paper reports improvements under both matched token budgets and matched training compute, providing a more informative assessment of the method's effectiveness.

Critical weaknesses

W1. Insufficient comparison with closely related methods. The paper does not adequately distinguish ATAT from existing formulations such as GenMD4 and OeMDM/LoMDM, which already explore token-dependent masking schedules and generation ordering. Direct experimental comparisons are also missing.

W2. Limited experimental validation. Experiments are restricted to a relatively small training budget (0.98B tokens), and generation quality is not evaluated using Gen.PPL or MAUVE. This limits the evidence for the method's scalability and practical effectiveness.

W3. Potential issues in the theoretical justification. Lemma 7 appears to overlook cross-position dependencies in the denoising loss, raising concerns about the validity of its change-of-measure bound and the subsequent Theorem 6.

W4. Unclear relationship between token importance and likelihood optimization. High surprisal does not necessarily imply semantic importance, and the theoretical connection between importance-dependent masking and improvements in data likelihood remains insufficiently established.

Questions for authors

Q1. Novelty relative to prior formulations. GenMD4 already introduces token-dependent masking schedules and derives the corresponding forward/reverse processes and continuous-time ELBO. OeMDM/LoMDM further formulate diverse generation orders and jointly learn context-dependent ordering and denoising. How does ATAT's importance-aware masking and time-warping differ theoretically and methodologically from these approaches? Can similar strategies be implemented within existing formulations? Please provide direct comparisons with GenMD4 and LoMDM under matched training conditions.

Q2. Insufficient generation evaluation and training scale. All evaluations rely on NELBO-derived PPL, without generation metrics such as Gen.PPL or MAUVE. The effect of importance-aware decoding on generation quality and diversity also remains unverified. Comparisons with standard MDLM at matched NFEs are needed. Moreover, training uses only 0.98B tokens from OWT, substantially fewer than commonly used in prior work. Without longer training runs and generation-quality evaluations, the empirical evidence remains insufficient.

Q3. Validity of the change-of-measure bound in Lemma 7. Lemma 7 (Eq. 16) bounds the difference in expected denoising loss using per-position marginal total variation, yielding (C_{\max}\sum_l |m_l-\bar m|). However, each token's loss, (-\log p_\theta(x_l\mid z_t)), depends on the entire masking pattern, not just its own masking state. Changes in other positions can therefore affect the loss, making the marginal-TV bound questionable. How is this bound justified under cross-position dependencies? If it does not hold, how should the likelihood bound in Theorem 6 be revised?

Q4. Token importance and its connection to likelihood. ATAT uses EMA-teacher surprisal as a proxy for token importance, but difficult-to-predict tokens are not necessarily semantically important or useful as anchors for reconstructing other tokens. What evidence validates this assumption? Furthermore, ATAT's masking distribution (q_{\mathrm{ATAT}}(z_t\mid x)) depends on clean-token importance, yet training optimizes the uniform NELBO integrand under this nonuniform distribution rather than the exact ATAT NELBO. What theoretical or empirical evidence supports the claim that this objective improves the original data likelihood?

Reviewer_UrcTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

ATAT (btw it is unclear why it is called as it) is a importance-aware token masking strategy for training discrete diffusion models. it combines the token frequency in the corpus with a teacher model's importance scoring. The teacher is an EMA copy of the model under training. The metric used for evaluation is NELBO-NLL.

Critical strengths

Given that training diffusion models generally is more expensive and less efficient than AR models, investigations on how to design masking and how to decode are very centric to the core mechanisms of diffusion models.

Compared to random masking, it is natural to exploit more information oriented masking. The work reported some finding along this line.

Critical weaknesses
  1. Limited practical validations, NELBO-NLL alone is not sufficient.
  2. Importance scoring: using a teacher model and the specific prediction head, both settings are coarse proxies. Furthermore, that could be other ways to tell token importances, e.g., separation between information carrying and filler words. If the authors primarily propose the importance based masking strategy, it would make the contribution stronger with different importance estimation schemes, as the teacher model based one is neither simple enough nor particularly accurate.
  3. The method seems to require considerable more training overhead. This part analysis is not sufficient.
Questions for authors
  1. double check pseudo code and algorithm equations. they dont seem to be consistent
  2. Lemma 7, it appears to me that the predicting loss omits dependence with contextual masking. While for the same token, the masking probability stay the same, the prediction uncertainty depends on other tokens being masked or not. Lemma 7 oversimplified that and thus may subsequently invalidate Theorem 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.