- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper formulates MDLM training as a weighted denoising risk over masking rates and proposes NoNoMoAware (N2M), an adaptive timestep sampler. N2M tracks the second moment of the weighted loss in timestep bins, fits a Beta distribution to the estimated profile, and applies importance weights to preserve the expected risk. Experiments cover LLaDA and ModernBERT pretraining, supervised fine-tuning, and sampling ablations. The smaller tuning experiments use multiple random seeds, while the main pretraining comparisons use a single seed.
Critical strengths
- The method is straightforward, with low overhead from maintaining scalar statistics and fitting Beta parameters. Importance weighting preserves the target risk in expectation.
- The evaluation covers both pretraining and SFT, and includes comparisons with fixed Beta distributions and alternative fitting procedures. The results show improvements on several downstream tasks.
Critical weaknesses
-
Assumption 4.1 requires . However, Appendix B.1 states that , while the per-example weighted loss contains . Under a non-degenerate token cross-entropy, its second moment is typically as , which is not integrable at zero. The main Beta-sampling procedure does not specify a lower cutoff for ; the clipping mentioned in the appendix applies to the Gaussian-sampling baseline. Thus, the stated results about the variance under uniform sampling and the admissibility of the proposal are not supported by the described implementation.
-
The N2M row in Table 5 (MMLU-29.38/ Hella.-38.34/ ARC-c-32.64) conflicts with the corresponding results in Tables 2 and 10 (MMLU-49.38/Hella.-58.34/ARC-c-62.64), suggesting a reporting error. This is not a minor formatting discrepancy: it makes it difficult to determine which results are correct and substantially undermines confidence in the reliability of the paper's reported empirical results. Unless this is a misunderstanding, the authors need to explain the discrepancy and verify the affected tables and results.
-
The empirical evaluation is built primarily around masked diffusion training, especially LLaDA, with ModernBERT as an additional backbone. Block diffusion language models, including recent models such as LLaDA2.X and SDAR, are an increasingly relevant paradigm with promising practical applications, but are not evaluated. Since blockwise training and corruption may produce different timestep-loss profiles, the current experiments do not establish that the proposed sampler transfers to this setting.
Questions for authors
-
Does the actual training implementation impose a lower cutoff on ? If so, please report the cutoff, explain how the importance weights are handled, and discuss its effect on the objective and variance. If not, how do you address the possible growth of near zero? How do the fitted Beta parameters ensure that is finite?
-
Could you verify whether the N2M values in Table 5 are correct?
-
Why does the paper not evaluate N2M on block diffusion language models such as LLaDA2.X or SDAR? Do the timestep-loss estimates and importance-sampling correction remain applicable under blockwise corruption and training? An evaluation in this setting would substantially strengthen the evidence for the method's scope and practical relevance.
2 comments
Table 5
We thank the reviewer for catching this error, and we agree that it is not a minor one. The N2M row of Table 5 contained typos in three entries. It should read 49.38 / 58.34 / 62.64 / 25.93, the same values as the N2M row of Tables 2, 4 and 10. We have corrected this in the updated manuscript.
Blockwise training and corruption
We appreciate the suggestion.
The time-step loss estimates and the importance-sampling correction remain applicable to block diffusion language models, because block diffusion changes the attention structure and not the masking mechanism. One design choice is specific to this setting: time-steps are sampled independently per block instead of once per sample.
We did not evaluate LLaDA2.X or SDAR directly because of the lack of publicly available training recipes. Instead, we follow the block diffusion approach of Arriola et al. (2025) and fine-tune the auto-regressive Qwen3-0.6B model with block diffusion, with and without N2M. The table reports results after SFT on a mixture of allenai/tulu-3-sft-mixture, HuggingFaceTB/smoltalk, OpenCoder-LLM/opc-sft-stage1[lang:python], OpenCoder-LLM/opc-sft-stage2[lang:python]
| Model | GSM8K | BBH | MATH | MMLU | MMLU-Pro | HellaSwag | HumanEval | MBPP |
|---|---|---|---|---|---|---|---|---|
| Block diffusion | 46.3 | 26.6 | 12.9 | 39.1 | 13.8 | 39.3 | 46.3 | 38.2 |
| Block diffusion + N2M | 46.7 | 27.1 | 13.3 | 39.4 | 14.1 | 39.6 | 46.6 | 38.4 |
N2M also improves on the baseline here, on all eight benchmarks, which suggests that the method carries over to the block diffusion setting.
We acknowledge this concern. With the linear schedule, the number of masked tokens is Kt∼Binomial(L,t) and the per-example loss is ℓθ=t−1∑maskedCE. Suppose the cross-entropy at a masked position lies between m0>0 (the loss floor already used in Prop. A.1) and B (bounded logits). Using E[Kt2]=Lt(1−t)+L2t2,
m02(L2+tL(1−t))≤Vθ(t)≤B2(L2+tL(1−t)).
So on (0,1), Vθ(t) is Θ(1/t) as t→0 and is not integrable. Assumption 4.1 as stated cannot hold for a non-degenerate model, and by Lemma 3.7 the standard uniform-sampling estimator has infinite variance there as well. The paper stated its results on [0,1] without saying that the implementation clips t. We correct this below; the only change is the reference distribution of t.
Because tokens are discrete, a masking rate is informative only if it masks at least one token and leaves at least one unmasked. Every sampled time-step is therefore clipped to [ε,1−ε] with ε≥1/L, for uniform sampling and for N2M alike, so that at least one masked token is expected at the lower end and at least one unmasked token at the upper end. With L=2048 and ε=10−3, about two tokens are masked in expectation at t=ε and about two are unmasked at t=1−ε, which stabilizes training. Outside this range the loss is problematic. For t<1/L, a draw often masks no token and contributes zero loss, while a draw that masks a single token carries the weight 1/t; this is the regime that produces the 1/t growth identified by the reviewer. For t>1−1/L, the denoiser often has no unmasked context at all.
Our statements still hold once the clipping is taken into account. Let ρε denote the distribution of a clipped uniform draw. The proofs only assume that the samples are i.i.d. Thus we can re-write "t uniform on [0,1]" as "t distributed as ρε": integrals ∫01(⋅)dt become expectations over ρε, and the risk becomes
Rε(θ)=EXEt∼ρε[λ(t)hθ(X,t)].
Clipping has two effects:
For N2M, the Beta draw is clipped in the same way. The exact weight is the ratio of the clipped target to the clipped proposal: 1/p^(t) for a draw that is not clipped, as in Eq. 23, and the ratio of the endpoint masses, ε/F^(ε) or ε/(1−F^(1−ε)), for a draw clipped to the lower or upper end, where F^ is the CDF of the fitted Beta. Our implementation applies 1/p^(t) at the clipped value in the latter case as well. This differs from the exact ratio only at the two endpoints, whose total target mass is 2ε, so for the fitted values in Figure 2 the estimated risk differs from Rε by a term of order ε.
Up to this term, Lemma 4.2 gives an unbiased estimator of Rε. Because μ^,ν^>0, the Beta density has a positive minimum on [ε,1−ε]. The weight is therefore bounded by some wmax, and the second moment of the corrected estimator is at most wmaxB2(L2+L/ε), which is finite for every fitted (μ^,ν^). So it is the clipping that guarantees admissibility, and not the fitted parameters or the condition a,b>−2 of Assumption 4.7.
We will state the clipping and the form of the weight in Section 4.3, define the risk as Rε, and replace Assumption 4.1 by a lemma containing the two-sided bound above.