acceptodds
Under review as a conference paper at ICLR 2027

Watch Your (Time-)Step: Adaptive Sampling Helps Train Your Diffusion Language Model

Abstract

The success of _auto-regressive language models_ has spurred interest in _diffusion language models_, which can generate multiple tokens in parallel. Recent masked-diffusion language models often reuse bidirectional denoisers familiar from masked language modeling, but differ in how they distribute training across corruption levels. We focus on one common assumption in these models: uniform time-step sampling for noising sequences. We first view _masked diffusion language modeling_ (MDLM) as risk estimation over a continuum of fixed-rate _masked language modeling_ (MLM) denoising objectives. This perspective explains both why a fixed masking rate can be optimization-friendly, since it repeatedly trains the same corruption level, and why MDLM requires ELBO-dependent weights to estimate the full diffusion objective without bias. The same analysis shows that weighted losses can have time-step-dependent second moments, making uniform time-step sampling statistically inefficient. Motivated by this, we propose _**No**rmalized **2**nd **Mo**ment **Aware**ness_ (**NoNoMoAware**) sampling, an adaptive strategy that maintains online estimates of the weighted-loss second moment, fits a proposal over time-steps, and corrects with importance weights to preserve unbiased loss estimation. For empirical validation, we conduct experiments using different backbones during both pre-training and instruction-tuning, with our method showing downstream improvements on a number of different benchmarks. Our findings illustrate the need for more principled investigation into the training dynamics of DLMs and offer practical guidance for training such models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Models, Large Language Models, Masked Diffusion Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sJiqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates MDLM training as a weighted denoising risk over masking rates and proposes NoNoMoAware (N2M), an adaptive timestep sampler. N2M tracks the second moment of the weighted loss in timestep bins, fits a Beta distribution to the estimated profile, and applies importance weights to preserve the expected risk. Experiments cover LLaDA and ModernBERT pretraining, supervised fine-tuning, and sampling ablations. The smaller tuning experiments use multiple random seeds, while the main pretraining comparisons use a single seed.

Critical strengths
  1. The method is straightforward, with low overhead from maintaining scalar statistics and fitting Beta parameters. Importance weighting preserves the target risk in expectation.
  2. The evaluation covers both pretraining and SFT, and includes comparisons with fixed Beta distributions and alternative fitting procedures. The results show improvements on several downstream tasks.
Critical weaknesses
  1. Assumption 4.1 requires ∫01Vθ(t) dt<∞\int_0^1 V_\theta(t)\,dt<\infty. However, Appendix B.1 states that Kt∼Binomial(L,t)K_t\sim\mathrm{Binomial}(L,t), while the per-example weighted loss contains t−1∑maskedCEt^{-1}\sum_{\mathrm{masked}}\mathrm{CE}. Under a non-degenerate token cross-entropy, its second moment is typically Θ(1/t)\Theta(1/t) as t→0t\to0, which is not integrable at zero. The main Beta-sampling procedure does not specify a lower cutoff for tt; the [ϵ,1−ϵ][\epsilon,1-\epsilon] clipping mentioned in the appendix applies to the Gaussian-sampling baseline. Thus, the stated results about the variance under uniform sampling and the admissibility of the proposal are not supported by the described implementation.

  2. The N2M row in Table 5 (MMLU-29.38/ Hella.-38.34/ ARC-c-32.64) conflicts with the corresponding results in Tables 2 and 10 (MMLU-49.38/Hella.-58.34/ARC-c-62.64), suggesting a reporting error. This is not a minor formatting discrepancy: it makes it difficult to determine which results are correct and substantially undermines confidence in the reliability of the paper's reported empirical results. Unless this is a misunderstanding, the authors need to explain the discrepancy and verify the affected tables and results.

  3. The empirical evaluation is built primarily around masked diffusion training, especially LLaDA, with ModernBERT as an additional backbone. Block diffusion language models, including recent models such as LLaDA2.X and SDAR, are an increasingly relevant paradigm with promising practical applications, but are not evaluated. Since blockwise training and corruption may produce different timestep-loss profiles, the current experiments do not establish that the proposed sampler transfers to this setting.

Questions for authors
  1. Does the actual training implementation impose a lower cutoff on tt ? If so, please report the cutoff, explain how the importance weights are handled, and discuss its effect on the objective and variance. If not, how do you address the possible 1/t1/t growth of Vθ(t)V_\theta(t) near zero? How do the fitted Beta parameters ensure that ∫Vθ(t)/p(t) dt\int V_\theta(t)/p(t)\,dt is finite?

  2. Could you verify whether the N2M values in Table 5 are correct?

  3. Why does the paper not evaluate N2M on block diffusion language models such as LLaDA2.X or SDAR? Do the timestep-loss estimates and importance-sampling correction remain applicable under blockwise corruption and training? An evaluation in this setting would substantially strengthen the evidence for the method's scope and practical relevance.

2 comments

Assumption 4.1

We acknowledge this concern. With the linear schedule, the number of masked tokens is Kt∼Binomial(L,t)K_t \sim \mathrm{Binomial}(L, t) and the per-example loss is ℓθ=t−1∑maskedCE\ell_\theta = t^{-1} \sum_{\mathrm{masked}} \mathrm{CE}. Suppose the cross-entropy at a masked position lies between m0>0m_0 \gt 0 (the loss floor already used in Prop. A.1) and BB (bounded logits). Using E[Kt2]=Lt(1−t)+L2t2\mathbb E [K_t^2] = L t (1-t) + L^2 t^2,

m02(L2+L(1−t)t)≤Vθ(t)≤B2(L2+L(1−t)t).m_0^2 \Big( L^2 + \frac{L(1-t)}{t} \Big) \le V_\theta(t) \le B^2 \Big( L^2 + \frac{L(1-t)}{t} \Big).

So on (0,1)(0,1), Vθ(t)V_\theta(t) is Θ(1/t)\Theta(1/t) as t→0t \to 0 and is not integrable. Assumption 4.1 as stated cannot hold for a non-degenerate model, and by Lemma 3.7 the standard uniform-sampling estimator has infinite variance there as well. The paper stated its results on [0,1][0,1] without saying that the implementation clips tt. We correct this below; the only change is the reference distribution of tt.

Because tokens are discrete, a masking rate is informative only if it masks at least one token and leaves at least one unmasked. Every sampled time-step is therefore clipped to [ε,1−ε][\varepsilon, 1-\varepsilon] with ε≥1/L\varepsilon \ge 1/L, for uniform sampling and for N2M alike, so that at least one masked token is expected at the lower end and at least one unmasked token at the upper end. With L=2048L = 2048 and ε=10−3\varepsilon = 10^{-3}, about two tokens are masked in expectation at t=εt = \varepsilon and about two are unmasked at t=1−εt = 1-\varepsilon, which stabilizes training. Outside this range the loss is problematic. For t<1/Lt \lt 1/L, a draw often masks no token and contributes zero loss, while a draw that masks a single token carries the weight 1/t1/t; this is the regime that produces the 1/t1/t growth identified by the reviewer. For t>1−1/Lt \gt 1 - 1/L, the denoiser often has no unmasked context at all.

Our statements still hold once the clipping is taken into account. Let ρε\rho_\varepsilon denote the distribution of a clipped uniform draw. The proofs only assume that the samples are i.i.d. Thus we can re-write "tt uniform on [0,1][0,1]" as "tt distributed as ρε\rho_\varepsilon": integrals ∫01(⋅)dt\int_0^1 (\cdot) dt become expectations over ρε\rho_\varepsilon, and the risk becomes

Rε(θ)=EXEt∼ρε[λ(t)hθ(X,t)].R_\varepsilon(\theta) = \mathbb E_X \mathbb E_{t \sim \rho_\varepsilon} [ \lambda(t) h_\theta(X, t) ].

Clipping has two effects:

  1. The target changes by a term of order ε\varepsilon. Clipping moves tt in [0,ε)[0, \varepsilon) and (1−ε,1](1-\varepsilon, 1] to the nearest endpoint. Since λ(t)EX[hθ(X,t)]=Lcθ(t)\lambda(t) \mathbb E_X [h_\theta(X,t)] = L c_\theta(t), where cθ(t)≤Bc_\theta(t) \le B is the mean cross-entropy of a masked token at masking rate tt, we have ∣R(θ)−Rε(θ)∣≤2εLB\lvert R(\theta) - R_\varepsilon(\theta) \rvert \le 2 \varepsilon L B. The target RεR_\varepsilon is the same for N2M and for the uniform baselines.
  2. The second moment becomes bounded. On the clipped range the bound above gives Vθ(t)≤B2(L2+L/ε)V_\theta(t) \le B^2 (L^2 + L/\varepsilon), and since ε≥1/L\varepsilon \ge 1/L this is at most 2B2L22 B^2 L^2. Requiring at least one masked token in expectation is what keeps the 1/t1/t term from dominating. With L=2048L = 2048 and ε=10−3\varepsilon = 10^{-3}, that term is 0.49 times the regular term L2L^2 at t=εt = \varepsilon and below 5 percent of it for t≥0.01t \ge 0.01. Assumption 4.1 then follows from bounded cross-entropy and no longer needs to be assumed, and the variances in Lemma 3.7 and Lemma 4.2 are finite. When fewer tokens can be masked, as for SFT responses, ε≥1/L\varepsilon \ge 1/L may not hold, but the first bound still applies and the conclusions are unchanged.

For N2M, the Beta draw is clipped in the same way. The exact weight is the ratio of the clipped target to the clipped proposal: 1/p^(t)1/\hat p(t) for a draw that is not clipped, as in Eq. 23, and the ratio of the endpoint masses, ε/F^(ε)\varepsilon / \hat F(\varepsilon) or ε/(1−F^(1−ε))\varepsilon / (1 - \hat F(1-\varepsilon)), for a draw clipped to the lower or upper end, where F^\hat F is the CDF of the fitted Beta. Our implementation applies 1/p^(t)1/\hat p(t) at the clipped value in the latter case as well. This differs from the exact ratio only at the two endpoints, whose total target mass is 2ε2\varepsilon, so for the fitted values in Figure 2 the estimated risk differs from RεR_\varepsilon by a term of order ε\varepsilon.

Up to this term, Lemma 4.2 gives an unbiased estimator of RεR_\varepsilon. Because μ^,ν^>0\hat\mu, \hat\nu \gt 0, the Beta density has a positive minimum on [ε,1−ε][\varepsilon, 1-\varepsilon]. The weight is therefore bounded by some wmax⁡w_{\max}, and the second moment of the corrected estimator is at most wmax⁡B2(L2+L/ε)w_{\max} B^2 (L^2 + L/\varepsilon), which is finite for every fitted (μ^,ν^)(\hat\mu, \hat\nu). So it is the clipping that guarantees admissibility, and not the fitted parameters or the condition a,b>−2a, b \gt -2 of Assumption 4.7.

We will state the clipping and the form of the weight in Section 4.3, define the risk as RεR_\varepsilon, and replace Assumption 4.1 by a lemma containing the two-sided bound above.

Table 5

We thank the reviewer for catching this error, and we agree that it is not a minor one. The N2M row of Table 5 contained typos in three entries. It should read 49.38 / 58.34 / 62.64 / 25.93, the same values as the N2M row of Tables 2, 4 and 10. We have corrected this in the updated manuscript.

Blockwise training and corruption

We appreciate the suggestion.

The time-step loss estimates and the importance-sampling correction remain applicable to block diffusion language models, because block diffusion changes the attention structure and not the masking mechanism. One design choice is specific to this setting: time-steps are sampled independently per block instead of once per sample.

We did not evaluate LLaDA2.X or SDAR directly because of the lack of publicly available training recipes. Instead, we follow the block diffusion approach of Arriola et al. (2025) and fine-tune the auto-regressive Qwen3-0.6B model with block diffusion, with and without N2M. The table reports results after SFT on a mixture of allenai/tulu-3-sft-mixture, HuggingFaceTB/smoltalk, OpenCoder-LLM/opc-sft-stage1[lang:python], OpenCoder-LLM/opc-sft-stage2[lang:python]

ModelGSM8KBBHMATHMMLUMMLU-ProHellaSwagHumanEvalMBPP
Block diffusion46.326.612.939.113.839.346.338.2
Block diffusion + N2M46.727.113.339.414.139.646.638.4

N2M also improves on the baseline here, on all eight benchmarks, which suggests that the method carries over to the block diffusion setting.

Reviewer_o9UFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether uniform timestep sampling allocates training examples efficiently across the corruption levels learned by masked diffusion language models. It distinguishes changing the denoising objective, as in fixed-rate MLM, from improving estimation of the same MDLM objective. The analysis relates scalar estimator variance to the timestep-dependent second moment of the weighted denoising loss. The proposed method adapts timestep sampling using online loss statistics and applies importance correction to preserve the objective, using a fitted Beta distribution to implement the proposal. Experiments on LLaDA and ModernBERT evaluate pretraining and supervised fine-tuning, including repeated hyperparameter comparisons and additional adaptive-sampling baselines. The results support the practical usefulness of revisiting timestep allocation in MDLM training.

Critical strengths
  1. The paper identifies a useful training-efficiency question while preserving the MDLM objective. Its central question is whether conventional allocation across corruption levels introduces avoidable estimation noise. The analysis separates this issue from choosing a fixed masking rate or removing the ELBO weights, both of which change the objective. This provides a coherent motivation for adapting the sampler and a practical way to improve training without changing the denoiser architecture. I consider the diagnosis and validation of this opportunity in MDLM a meaningful contribution.

  2. The experiments provide positive evidence across training stages, including repeated comparisons. Both LLaDA and ModernBERT are evaluated in pretraining and SFT. Table 1 includes three-seed pretraining results for the subsequently adopted 20-bin configuration, with an average score of 40.48 versus 39.66 for uniform sampling. Appendix C.5 provides a five-seed comparison against direct binned adaptive sampling: MMLU improves from 30.66 ± 0.66 with uniform sampling to 31.48 ± 0.54 with the binned sampler and 32.54 ± 0.75 with the proposed implementation. Together with SFT results across datasets, these experiments support the practical usefulness of adaptive timestep allocation.

Critical weaknesses
  1. The empirical connection between the diagnosed estimation inefficiency and optimization improvement is incomplete. The analysis explains when uniform sampling fails to minimize scalar estimator variance, but the diagnostics do not clearly quantify how much additional noise this introduces in actual MDLM training or how much the sampler removes. Appendix C.6 presents its plots as evidence for gradient-estimator variance claims, whereas Figure 6 reports average per-bin scalar loss variance. This directly measures neither the complete importance-corrected estimator variance in Equation 14 nor gradient variance; Proposition 4.6 establishes an upper bound under an additional assumption. The performance gains remain useful evidence, but this mismatch limits how strongly the paper establishes its explanation of why training improves. Clarifying the measurement and narrowing the interpretation, or providing a directly corresponding measurement, would address this concern.

  2. The main pretraining improvements are modest in magnitude. In Table 2, the average downstream score increases from 45.38 to 45.90 for LLaDA and from 44.20 to 44.50 for ModernBERT, gains of 0.52 and 0.30 percentage points. Table 1 provides positive three-seed evidence for a larger 0.82-point improvement with the selected 20-bin setting, and SFT experiments provide further support. These gains can be useful for a lightweight modification, but the evidence for a substantial improvement in backbone training remains limited. A common evaluation of the original MDLM objective at matched training budgets would help assess whether the small downstream gains accompany a clearer improvement in optimization. This concern is about the magnitude and interpretation of the benefit.

Questions for authors
  1. Evaluation under the original MDLM objective: After training with the modified sampler, have the authors evaluated the frozen model using the original weighted MDLM loss under a common uniform-timestep validation protocol? Comparing uniform-trained and adaptively trained checkpoints at matched training budgets, using the same held-out examples and corruption realizations, and reporting the mean and variance across evaluation minibatches would help establish whether model quality and loss variability improve under the same evaluation conditions. This retains the original ELBO weight and requires no adaptive inverse-density correction because evaluation samples timesteps uniformly. As a separate check of the sampling mechanism, the authors could hold the checkpoint fixed and compare the complete corrected-estimator variance under the two samplers, freezing the adaptive proposal and using fresh evaluation samples. Please clarify how Figures 5–6 relate to these measurements. These are suggested analyses of existing checkpoints, without additional training runs.

  2. Applicability to block diffusion: Have the authors evaluated the method in block-diffusion training, and is there any existing evidence of its effectiveness in that setting? Conditioning on preceding blocks changes the available context and may change the timestep-dependent loss statistics. If this setting has not been evaluated, please clarify whether the proposed sampling and importance correction apply directly to its training objective and what limitations are expected. This is a question about applicability and existing evidence; a new block-diffusion training study would be an extension.

  3. Representation quality — optional supporting analysis: Section 5.2 interprets the pretraining results as improved representation quality. Is there evidence that the gains also appear when the backbone is frozen and evaluated as a feature extractor? A small linear-probe comparison between uniform-trained and adaptively trained checkpoints, with matched input corruption, layer, pooling, and probe-training settings, would help assess the transferability of the learned features. This is an optional analysis using existing checkpoints, with only the probe trained; it would support the representation-quality interpretation rather than establish the variance-reduction mechanism.

3 comments

The empirical connection between the diagnosed estimation inefficiency and optimization improvement is incomplete

Evaluation under the original MDLM objective

We agree with this assessment. Appendix C.6 measures the following:

  • Figure 5 is the training loss of each run along its own trajectory. For N2M this is the importance-corrected loss, so both curves estimate the same objective. It is not a held-out evaluation.
  • Figure 6 is, for each run along its own trajectory, the variance of the weighted per-example loss within each time-step bin, averaged over bins. As the reviewer says, this is neither the variance of the importance-corrected estimator in Eq. 14 nor a gradient variance. The two curves are also measured at different parameters, so the figure mixes the effect of the sampler with the state of the model. We now describe Figure 6 as a loss-variance diagnostic.

Following the reviewer's suggestion, we ran the proposed analyses on existing checkpoints, with no additional training.

Common-protocol evaluation of the original objective. We evaluate checkpoints on the same held-out sequences with the same time-step and mask realizations, with tt drawn uniformly and no inverse-density correction. We report the mean ± standard deviation of per-token losses across 50 evaluation mini-batches.

Uniform-trainedN2M-trained
3.62 ± 0.742.81 ± 0.61

Variance of the sampling mechanism at a fixed checkpoint. With the parameters frozen and the proposal frozen at the Beta fitted at that checkpoint (μ=2.101,ν=1.472\mu=2.101, \nu=1.472), we estimate (i) the variance of the complete importance-corrected estimator, N⋅Var(R^N,p)N \cdot \mathrm{Var}(\hat R_{N,p}), for the uniform density and for the fitted proposal, which is the quantity in Eq. 14, and (ii) the gradient variance E∥g−gˉ∥22\mathbb E\lVert g - \bar g \rVert_2^2, which is the left-hand side of Eq. 18.

QuantityUniformFitted proposal
Estimator variance (Eq. 14)3.863.34
Gradient variance (Eq. 18)4.343.85

This confirms our analysis that using an adaptive weighting approach can enable empirical variance reduction.


The main pretraining improvements are modest in magnitude

We acknowledge that the demonstrated gains are modest, and we have tried to present them as such so as not to overstate the evidence. The evidence is:

  • Table 2 (single seed): the average improves by 0.52 points for LLaDA and 0.30 for ModernBERT; N2M is at or above uniform sampling on 11 of 12 tasks for LLaDA and 9 of 12 for ModernBERT.
  • Table 1 (three seeds, the 20-bin configuration): 40.48 against 39.66, a gain of 0.82.
  • Table 3: using N2M in both pre-training and SFT improves the average over uniform sampling in all six backbone and dataset combinations, by 0.48 to 0.88 points.
  • Appendix C.5 (five seeds): N2M is the best of the three samplers on 8 of 9 tasks.

Our emphasis is on a consistent improvement that comes at negligible cost: the method changes no other training choice and adds only a few hyper-parameters and scalar loss statistics that are maintained online. If any wording still appears overstated, we are happy to revise it.

We also agree that the common-protocol evaluation above is the right test of whether the downstream gains accompany a clearer improvement in optimization. The training budgets are matched: all methods in Table 2 see the same number of tokens during training. In addition, Table 10 in Appendix C.4 shows that N2M is more effective than indiscriminately using high masking rates, a comparison in which the number of masked tokens seen, and thus the number of tokens on which the loss is computed, is more consistent between methods.


Applicability to block diffusion

We appreciate the suggestion.

The method remains applicable to block diffusion language models, because block diffusion changes the attention structure and not the masking mechanism. One design choice is specific to this setting: time-steps are sampled independently per block instead of once per sample.

We did not evaluate LLaDA2.X or SDAR directly because of the lack of publicly available training recipes. Instead, we follow the block diffusion approach of Arriola et al. (2025) and fine-tune the auto-regressive Qwen3-0.6B model with block diffusion, with and without N2M. The table reports results after SFT on a mixture of allenai/tulu-3-sft-mixture, HuggingFaceTB/smoltalk, OpenCoder-LLM/opc-sft-stage1[lang:python], OpenCoder-LLM/opc-sft-stage2[lang:python]

ModelGSM8KBBHMATHMMLUMMLU-ProHellaSwagHumanEvalMBPP
Block diffusion46.326.612.939.113.839.346.338.2
Block diffusion + N2M46.727.113.339.414.139.646.638.4

N2M also improves on the baseline here, on all eight benchmarks, which suggests that the method carries over to the block diffusion setting.

Representation quality

We acknowledge that the sentence in Section 5.2 stating that N2M "improves the quality of the learned representation" was meant as shorthand for improved downstream performance after pre-training, which may have caused confusion. We will rewrite it to refer explicitly to downstream performance as the primary claim.

Thank you for the detailed response and the additional analyses. The common-protocol loss evaluation and fixed-checkpoint variance measurements substantially address my concern that the original diagnostics did not directly support the claimed variance-reduction mechanism. I also appreciate the additional block-diffusion experiment and the narrowing of the representation-quality claim. I acknowledge the practical value of the performance gains from a lightweight training modification.

Following the discussion of Zheng et al. raised by reviewer QR8E and the authors’ response, my main remaining concern is the extent of the contribution beyond prior work. Second-moment-based adaptive sampling and importance correction have already been applied to masked-model training. I therefore view the contribution here as the MDLM-specific analysis and the pooled-bin, low-dimensional Beta proposal design. Appendix C.5 provides useful evidence for the additional benefits of this implementation. However, without a direct comparison with the specific scheme of Zheng et al., I remain unconvinced that these differences establish a sufficiently substantial advance over the closely related prior approach. If results from the comparison mentioned in your response are available, please share them together with the experimental setting.

I appreciate the improvement in the mean loss under the common evaluation protocol. I am also interested in whether this improvement occurs broadly across masking rates or is concentrated in particular regions. If feasible, could you break down the existing evaluation by timestep bin and compare the mean loss of the uniform-trained and N2M-trained checkpoints? A table or simple plot using the same evaluation examples, timesteps, and masks, while retaining the weighting and normalization of the reported evaluation, would help clarify where the improvement occurs.

Reviewer_QR8EOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes NoNoMoAware, an adaptive time-step sampling strategy for training masked diffusion language models (MDLMs). The paper interprets the MDLM objective as a weighted combination of denoising objectives with fixed masking rate and argues that uniformly sampling diffusion time steps may lead to inefficient loss estimation. Based on the second moment of the weighted denoising loss, the paper proposes to fit a Beta distribution to adaptively sample time steps while preserving the original training objective through importance weighting. NoNoMoAware is evaluated on different model backbones during pre-training and supervised fine-tuning, with reported improvements on several downstream benchmarks.

Critical strengths
  1. A practical approach to adaptive time-step sampling. The proposed method is relatively simple to implement and provides a principled motivation for non-uniform time-step sampling. Using importance weighting to preserve the original MDLM objective is an appealing property.
  2. Evaluation across different training settings. The authors evaluate the approach with multiple model backbones and during both pre-training and supervised fine-tuning. These experiments demonstrate the potential applicability of the method.
Critical weaknesses
  1. Unclear experimental setup. Some model configurations and implementation details are not clearly explained, making it difficult to understand and reproduce the experiments. Insufficient motivation and empirical evidence.
  2. The paper does not clearly explain why the risk-minimization perspective is necessary or provide enough evidence that the proposed method improves optimization efficiency.
  3. The idea of sampling diffusion time steps based on the second moment of the loss has been explored in previous work for example [1]. The novelty of the proposed approach needs to be clarified. And the paper should at least use [1] as a baseline to compare to.

[1] Zheng, Kaiwen, et al. "Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors
  1. Line 322-338: “In particular, we use a LLaDA (Nie et al., 2025b) model with approximately 400M non-embedding parameters trained on 10B tokens from Li et al. (2024) with a context length of 2048 tokens.” What does the author mean here. The original released LLaDA is a 8B model with 4096-token as context length. And the original LLaDA paper doesn’t release or specify their training data. Does the author refer to that they train a LLaDA-like model from scratch with NoNoMoAware instead of initializing from the pretrained LLaDA model?
  2. What are the metrics used in Table 4 and 5? I would assume acc, but then why are the N2MN^2M results much worse than the others?
  3. Why is it necessary to reformulate the MDLM objective as a risk-minimization problem? Does this formulation provide any additional theoretical insights that cannot be obtained directly from the original MDLM objective?
  4. The paper argues that uniform time-step sampling is inefficient for optimization. However, I did not find sufficient empirical evidence to support this claim. Could the authors provide more direct comparisons of optimization efficiency, such as gradient variance or convergence speed?
  5. Lemma 3.7: What does the superscript or subscript "unw" mean? I assume it stands for "unweighted," but this should be explained.
3 comments

Unclear experimental setup.

Line 322-338

We thank the reviewer for raising this point. Our description of the setup was not clear enough, and we have added the details below to the manuscript.

These models are trained from scratch and are not initialized from the released checkpoints. They are trained on 50B tokens of DCLM (Li et al., 2024) with a context length of 2048 tokens.

Because of limited computational resources, we use reduced LLaDA and ModernBERT configurations: a LLaDA configuration with about 1B parameters (the released model has 8B) and a ModernBERT configuration with about 360M parameters. We obtain them from the configurations of GSAI-ML/LLaDA-8B-Base and answerdotai/ModernBERT-large by changing the number of layers, the hidden size and the number of attention heads per layer.


The paper does not clearly explain why the risk-minimization perspective is necessary or provide enough evidence that the proposed method improves optimization efficiency.

Why is it necessary to reformulate the MDLM objective as a risk-minimization problem? Does this formulation provide any additional theoretical insights that cannot be obtained directly from the original MDLM objective?

We would like to clarify that the risk view does not introduce a new objective in and of itself. R(θ)R(\theta) in Eq. 6 is the MDLM NELBO, written as an expectation over data, time-steps and masks. We use this form because the ELBO specifies what is optimized but says nothing about how well a mini-batch estimates it, and the question we study (is uniform sampling of tt a good choice?) is a question about the estimator. Writing the objective as an expected loss over time-steps makes three observations explicit, and our analysis and the proposed correction build on them:

  1. A separation between changing the objective and changing the estimator. Fixing the masking rate (MLM) or dropping λ(t)\lambda(t) changes the target risk (Props. 3.3 to 3.5), whereas changing the sampling density of tt with importance correction does not (Lemma 4.2). This is what distinguishes N2M from the "MLM (50%)" and "No Weighting" baselines of Table 2.
  2. An exact variance decomposition (Lemma 3.7) into a within-time-step and a between-time-step term, which shows what fixed-rate MLM removes and at what cost.
  3. The variance-optimal proposal p⋆∝Vθp^\star \propto \sqrt{V_\theta} (Thm. 4.3) and the Beta parameterization that follows from it (Prop. 4.8). These identify the second moment of the weighted loss as the quantity that matters, which is why our method tracks it.

We agree that, at its core, this is a standard importance-sampling argument applied to MDLM. We do not claim the reformulation as a contribution in itself; its role is to make the approach we propose precise. We will state this explicitly in Section 3.

The idea of sampling diffusion time steps based on the second moment of the loss has been explored in previous work

We thank the reviewer for this reference and will discuss it in Section 6. The main contributions of [1] are the time-agnostic reformulation of the ELBO, the first-hitting sampler, and the analysis of numerical precision in categorical sampling. Most relevant to our work is its Appendix J.1.2 ("Failed Training Attempts"), where the adaptive importance sampler of Nichol and Dhariwal (2021) is applied over the number of masked tokens nn: the proposal is proportional to the square root of the second moment of the loss for each nn, estimated from the 10 most recent losses for that nn. The authors report that this reduces loss variance but degrades performance, and conjecture that the dynamically updated proposal biases the estimator.

We agree that sampling in proportion to the square root of the loss second moment is not new. Our contribution lies in the analysis of Section 3, which neither Nichol and Dhariwal (2021) nor [1] provide, and in two differences in approach that follow from it:

  1. A low-dimensional proposal in place of a tabular one. The scheme in [1] keeps a separate entry for each of the LL possible values of nn. In our setting (L=2048L = 2048, about 244 sequences per step) each entry would be refreshed about once every eight steps on average, from 10 samples. N2M pools all observations into T=20T = 20 bins and a two-parameter Beta fit, derived from a boundary-aware power law for VθV_\theta (Assumption 4.7, Prop. 4.8).
  2. An explicit treatment of adaptivity. Lemma 4.2 shows that if the proposal for the current batch is determined by previous batches only, the loss and gradient estimators remain unbiased conditionally on that history. This addresses the bias concern raised in [1] for our scheme.

In addition, Appendix C.5 already compares N2M with direct sampling from the binned estimate V^j\sqrt{\hat V_j}, which is the closest analogue of the tabular scheme in our codebase. Over 5 seeds, the binned sampler improves on uniform sampling on 8 of 9 tasks, and the Beta fit improves on the binned sampler on 8 of 9 tasks. We are also adding the scheme of [1] (one entry per nn, 10-sample history) as a baseline. To our knowledge, the code for that experiment has not been released, so we are reimplementing it as faithfully as we can.


What are the metrics used in Table 4 and 5? I would assume acc, but then why are the N2MN^2M results much worse than the others?

We thank the reviewer for catching this reporting error. Tables 4 and 5 both report accuracy. The N2M row of Table 5 contained typos: it should match the N2M row of Table 4, that is, 49.38 / 58.34 / 62.64 / 25.93. The lower values in Table 4 (the "+ Per Token Rate" rows) are intended. That ablation shows that assigning a separate time-step, and therefore a separate masking rate, to each token does not benefit learning compared with the standard single time-step per sample.


The paper argues that uniform time-step sampling is inefficient for optimization. However, I did not find sufficient empirical evidence to support this claim. Could the authors provide more direct comparisons of optimization efficiency, such as gradient variance or convergence speed?

Figures 5 and 6 in Appendix C.6 give some evidence: N2M reaches a lower training loss (Figure 5) and a lower per-bin loss variance (Figure 6). In addition, we measured the gradient variance E∥g−gˉ∥22\mathbb E \lVert g - \bar g \rVert_2^2, which is the left-hand side of Eq. 18, over 50 randomly sampled held-out mini-batches under both samplers. This uses the trained 1B LLaDA checkpoints and requires no retraining.

UniformN2M
Gradient variance (50 mini-batches)4.343.85

Regarding convergence speed, Figure 5 shows the training loss that each method reaches at each step. If the reviewer has a different measure of convergence in mind, we are happy to provide it.


Lemma 3.7: What does the superscript or subscript "unw" mean? I assume it stands for "unweighted," but this should be explained.

Yes, it stands for "unweighted". We have edited the manuscript to state this explicitly.

Thank you for the replies. Below are my comments: Using risk-minimization as a proxy to analyse the MDM ELBO sounds more reasonable and this explaination solves my major concern. In terms of the related work, it would be great if the authors could add experiments to compare to the methods sampling timesteps non-uniformly to show the soundness of the NoNoMoAware sampling. The right part of Figure 1 is confusing. How is the loss variance measured? What is the function of the xy-plane in the figure? In general I feel the quality of the updated paper improves substantially compared to the last version. I would raise my rating to 2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.