acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Accelerating the Training of Masked Diffusion Language Models

Abstract

Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.

Primary area:
generative models
Keywords:
Discrete diffusion models, language generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qjuJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors investigated why mask diffusion models learn slower than autoregressive models during pretraining phase. They attributed the inefficiency to locality bias, and proposed a Bell-shaped sampling strategy to improve the training efficiency.

Critical strengths
  1. the proposal is very simple to apply, where the reported gains look decent.
  2. the authors did a thorough experimental and algorithmic analysis to design the sampling mechanism.
Critical weaknesses
  1. The proposed recipe is basically about how often different masking ratios shall be used during training, rather than actual good masking patterns. High- and low-context are essentially avoided by not having low and high masking ratios, this has less dependence with masking patterns, i.e., the whole reasoning behind locality may just boil down to token masking ratio, where patterns do not matter so much
  2. while the work targeted pretraining efficiency and used common metrics, the real impact can only be validated after testing in post-training and down-stream tasks. In Table 4, the authors did some testing, would be interesting to extend that, e.g., zero-shot performance, and also comparison to AR model baselines.
Questions for authors
  1. is masking ratio the most effective mechanism to exploit locality? besides the current masking patterns, does it make more sense to design more locality oriented patterns, e.g., random contiguous spans and etc.
  2. on the ordering pattern side, what is new relative the prior work Xue et al. (2025), which already compares L2R, fixed-random, and blockwise-random orders?
  3. Table 4 suggested the proposed sampling benefits mostly during CPT phase, very marginal during post-training and with SFT. Any explanation?
3 comments
AuthorsOpenReview ↗

Author Rebuttal (1/3)

We thank the reviewer for the valuable feedback! We address the reviewer's questions below, and the changes in the revised paper are highlighted in blue.

W1. The proposed method adjusts masking ratios rather than directly optimizing masking patterns.

We thank the reviewer for this insightful comment. We would like to clarify that locality bias is the underlying factor motivating our analysis, whereas the inefficiency of low- and high-context samples is its consequence. Our method aims to mitigate this consequence rather than directly modify the locality bias of language.

More specifically, due to locality bias, low-context samples often lack informative nearby tokens and quickly converge to their optimum (Sec. 3.2), while high-context samples contain many distant observed tokens that contribute little additional learning signal (Sec. 3.3). This motivates reducing the frequency of these inefficient context regimes.

We agree that our method does not explicitly select individual masking patterns based on their locality. However, under the standard MDM forward process, the masking ratio directly controls the distribution of local masking patterns. In particular, Definition 4.3 and Proposition 4.4 formally establish that bell-shaped time sampling reduces the probability of long consecutive runs of masks or non-masks, which are motivated by the inefficiencies analyzed in Secs. 3.2 and 3.3.

Therefore, our contribution is not to identify optimal masking patterns individually, but to provide a simple way to reduce the occurrence of inefficient local-context configurations through time sampling.

Q1. Would more locality-oriented masking patterns, such as span masking, be more effective?

We thank the reviewer for this suggestion. We do not claim that masking-ratio adjustment is necessarily the most effective approach. However, span masking introduces consecutive masked tokens, which is contrary to our motivation of reducing long consecutive runs of masks or non-masks. Moreover, span masking modifies the standard independent-masking forward process, whereas our method preserves it.

Second, artificially adjusting masking patterns, such as span masking, might not preserve the optimum of the theoretical NELBO. For example, span masking alters the standard MDM forward process by replacing independent corruption across positions with dependent corruption, under which the standard diffusion formulation no longer applies. This contrasts with Proposition 4.2, which proves that bell-shaped time sampling preserves the same optimum as the theoretical NELBO. Thus, the optimizer-preservation guarantee in Proposition 4.2 does not directly extend to such modifications of the masking process.

To investigate this empirically, we additionally conducted experiments using the official SpanBERT-style masking strategy [1], sampling span lengths from Geo(0.2)\mathrm{Geo}(0.2) until reaching the desired masking ratio. The test NLLs after 0.5M training steps on LM1B are:

Base MDMBase MDM + Span MaskingOursOurs + Span Masking
NLL3.485.47 (+1.99)3.415.41 (+2.00)

Span masking substantially degrades performance for both the standard MDM and our method. While this does not rule out other locality-aware masking strategies, it suggests that simply introducing contiguous masked spans does not complement the training of masked diffusion models. We believe exploring more sophisticated locality-aware masking strategies is an interesting direction for future work; yet we hope reviewer will understand that we intended to provide simple and basic method that can accelerate the training of MDM. The details have been added to Appendix C.8 of the revised paper.

[1] Joshi et al., "SpanBERT: Improving Pre-training by Representing and Predicting Spans," Transactions of the association for computational linguistics, 2020.

W2. Provide broader downstream evaluation including zero-shot performance and comparison with ARMs.

We thank the reviewer for this valuable suggestion! Before presenting additional comparisons, we would like to clarify two points.

First, our paper already provides an extensive evaluation of MDM training efficiency and downstream performance. Specifically, we evaluate (1) validation NLL on three settings, (2) generative perplexity, (3) zero-shot perplexity on seven benchmarks, (4) downstream performance on eight benchmarks, and (5) instruction-following performance on two benchmarks. To our knowledge, this represents one of the more comprehensive evaluations of an MDM pretraining paper.

Second, our goal is to improve the training efficiency of existing MDMs, rather than to claim superiority over ARMs. Our primary comparisons therefore focus on standard MDM baselines. Nevertheless, we agree that comparisons with ARMs provide useful context, and we present additional results below.

AuthorsOpenReview ↗

Author Rebuttal (2/3)

(1) Zero-shot perplexity comparison with ARMs.

Although zero-shot perplexity results are already provided in Appendix C.4 (Table 8), we reproduce them below alongside the ARM baseline reported by Sahoo et al. [2] trained on same setting as ours.

TrainingPTBWikitextLM1BLambadaAG NewsPubmedArxiv
ARM82.0525.7551.2551.2852.0949.0141.73
Base115.9937.7672.0648.1868.3143.1537.85
Gaussian (Ours)98.1634.5167.7146.3961.7441.4337.88

Our method improves over the base MDM on six out of seven benchmarks. Although the ARM baseline performs better on several benchmarks, our method also achieves lower perplexity on Lambada, Pubmed, and Arxiv. These comparisons provide additional context for the performance gap between MDMs and ARMs, while confirming that our method consistently improves the standard MDM baseline.

(2) Comparison with GPT-2 Large (ARM) and additional zero-shot evaluation on unseen downstream tasks.

Following the reviewer's suggestion, we additionally compare our CPT+SFT models with their autoregressive initialization, GPT-2 Large (ARM), and extend the evaluation to ARC-E and BoolQ, two benchmarks not included in our SFT training data.

We clarify that the original downstream benchmarks in Table 4 are evaluated without in-context examples (zero-shot prompting), although their training splits were included in SFT. Therefore, we additionally evaluate ARC-E and BoolQ to assess generalization to tasks unseen during SFT.

TrainingCPTSFTLAMBADA (%)OBQA (%)Wino (%)PIQA (%)SIQA (%)InfillingTriQA (%)HellaSwag (%)ARC-E (%)BoolQ (%)
GPT-2 Large (AR)——40.4218.0050.3670.7837.4611.054.4535.6846.1460.95
Base + BaseBaseBase5.82 ± 0.1518.20 ± 0.2850.59 ± 0.6756.80 ± 0.2338.95 ± 0.809.87 ± 0.156.40 ± 0.1932.59 ± 1.2446.49± 0.2161.62± 1.33
Gaussian + GaussianGaussianGaussian17.89 ± 0.37 (+12.07)23.40 ± 1.41 (+5.20)49.80 ± 1.12 (-0.79)59.30 ± 0.88 (+2.50)42.07 ± 0.04 (+3.12)12.62 ± 0.38 (+2.75)27.80 ± 0.06 (+21.40)39.84 ± 1.21 (+7.25)50.88± 0.32 (+4.39)65.75± 1.27 (+4.13)

The results are presented in the table below. Gaussian time sampling improves over the standard MDM baseline on most benchmarks and outperforms GPT-2 Large on several tasks, including OBQA, SIQA, TriviaQA, and HellaSwag. More importantly, on the two unseen benchmarks, Gaussian consistently outperforms both the standard MDM baseline and GPT-2 Large. These additional results suggest that the benefits of Gaussian time sampling extend beyond the benchmarks used during SFT, providing further evidence of improved downstream generalization under our training setting.

We thank the reviewer again for encouraging us to further contextualize and strengthen our empirical evaluation.

[2] Sahoo et al., "Simple and Effective Masked Diffusion Language Models," NeurIPS'24.

AuthorsOpenReview ↗

Author Rebuttal (3/3)

Q2. Novelty relative to Xue et al. [3] in the generation-order analysis.

We thank the reviewer for raising this point. We have already discussed the distinction from prior ordering-based studies in the Related Work section. We appreciate this opportunity to further clarify our contributions.

Xue et al. [3] demonstrate that random generation orders lead to slower convergence than L2R orders, and suggest potential explanations for this observation. Our work goes beyond this finding in two important respects.

First, we investigate whether the difficulty of any-order learning primarily arises from order-space complexity or from the mismatch between random generation orders and the locality structure of language. Specifically, Sec. 3.1 compares σ\sigma-ARM, which learns a single fixed random order, with AoARM, which learns variable random orders. Despite the substantially larger order space encountered by AoARM, their convergence is comparable. This suggests that learning many different orders is not necessarily the primary source of training difficulty, and that locality-related mismatch may play a more important role.

Second, and more importantly, we investigate why locality-related effects lead to inefficient MDM training, rather than merely observing that random orders slow convergence. Specifically, Sec. 3.2 shows that low-context samples quickly converge to their optimum, leaving little reducible error despite their high loss. Sec. 3.3 further demonstrates that high-context samples contain redundant learning signals, as distant observed tokens contribute little additional gradient information. These analyses identify two distinct mechanisms underlying inefficient MDM training, which are not investigated by Xue et al.

Finally, these findings directly motivate our bell-shaped time sampling strategy, which reallocates training budget away from inefficient context regimes and substantially accelerates MDM training.

In summary, while Xue et al. establish that random generation orders hinder convergence, our contribution is to further investigate why this difficulty arises, characterize its consequences for MDM training, and develop a practical strategy to mitigate them. We have further clarified the distinction from Xue et al. in Appendix A.1 (Related Work) of the revised paper.

[3] Xue et al., "Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture", ICML'26.

Q3. Why are the improvements more pronounced during CPT than SFT?

We thank the reviewer for this thoughtful question. We agree that the improvements in Table 4 primarily arise from applying Gaussian time sampling during CPT, while its additional benefits during SFT are relatively modest.

We conjecture that this difference stems from the different roles and training durations of CPT and SFT. CPT involves 100K training steps, during which the model adapts from an ARM to an MDM and learns bidirectional denoising behavior. Thus, improving training efficiency through Gaussian time sampling can have a substantial cumulative effect. In contrast, SFT involves only 10K steps and starts from an already pretrained MDM, primarily adapting its learned representations to downstream tasks rather than learning general language modeling capabilities from scratch.

Nevertheless, we acknowledge that the relative benefits may also depend on the training objective, dataset, and optimization horizon. Our results suggest that the primary practical benefit of Gaussian time sampling arises during CPT, with these improvements largely carrying over to subsequent SFT.

We sincerely thank the reviewer again for detailed and thoughtful review. We would be happy to address any further questions or concerns.

Reviewer_5hW5OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks why masked diffusion models (MDMs) train more slowly than autoregressive models, and attributes this to the locality of language. It uses three analyses:

  • a comparison between autoregressive, fixed-random-order, and any-order (i.e. MDM) models during training (Fig. 2);
  • the MDM ELBO tracked during training for each number of visible context tokens (Fig. 3);
  • measurements of how a token's gradient changes when other nearby tokens are masked (Tables 1–2).

From these it argues that both low- and high-context training samples are inefficient.

The solution is to train on masked cross-entropy without the ELBO weight 1/t, with the masking rate t drawn from a Gaussian distribution around 0.5 (Eq. 6). With ~110M models, it reaches the 1M-step validation ELBO of the baseline ("Base", an MDM trained on the ELBO) 3.86×, 2.32× and 1.78× sooner on packed LM1B, unpacked LM1B and OpenWebText (OWT) (Fig. 6). Continual pretraining (CPT) and supervised fine-tuning (SFT) of GPT-2 Large also give downstream gains (Tables 4–5).

The paper is mostly clearly written, but two crucial points the results depend on are left unclear.

  • The sampler is not described: the unconditional, downstream, and instruction-following generations come with no generation algorithm, sampling step formula, step count, temperature or further detail. This matters because the choice and correct tuning of the sampler are crucial to maximize performance in discrete diffusion.
  • The main metric "NLL" is defined only in passing, towards the end of the paper (Sec. 5.1 identifies "NLL" as the bound of Eq. (1)). Figs. 2 and 7 say "training curve" for the same validation quantity. Important details such as units, the empirical estimator, and the LM1B validation split are not given.

On LM1B, unweighted masked cross-entropy at mid-range masking rates trains significantly faster than the negative ELBO. Mid-range masking rates also work better than reweighting toward them. These findings would be valuable if validated at language-modeling scale. At present, though, the mechanism and the largest speedup both come from packed LM1B, where packing shuffled sentences makes locality hold largely by construction, so it is hard to separate either from the dataset. A second large part of the speedup may come from dropping the 1/t weight alone, which the main comparisons change together with the sampling of t.

The ELBO measurements against the standard baseline look reliable. However, three problems remain:

  • the mechanism is tested only on packed LM1B, where locality largely holds by construction;
  • the main comparisons change two things at once (dropping the 1/t weight and drawing mid-range masking rates);
  • the evidence beyond LM1B is limited (OWT evidence is single-seed and not de-confounded from dropping the 1/t factor).

Resolving these would require additional multi-seed experiments on more representative data.

Critical strengths
  1. A simple change that costs nothing per step gives a large gain over the standard training recipe. The baselines in Fig. 6, trained on the ELBO with its 1/t weight, reproduce published MDLM results, and the method reaches their final ELBO much faster.
  2. The ablations separate weighting from sampling: the unweighted-uniform, reweighted and ELBO-preserving variants (Figs. 5 and 7) show that where t is sampled matters beyond the expected objective. On packed LM1B, the bell shape adds 1.5–2× even over unweighted uniform sampling, a stronger baseline than the ELBO.
Critical weaknesses
  1. The relevance of locality is not yet well supported, and the method is close to established practice. This matters because the paper's stated main contribution is explaining "why the two endpoint regimes are training-inefficient in MDMs" through mechanisms "fundamentally different from those identified in continuous diffusion" (App. A.1). However:

    • The evidence for locality comes only from packed LM1B, where shuffled sentences make distant tokens uninformative by construction, and the method itself is not local but only sets the global masking rate. At low noise, the paper's own results also point elsewhere: high-context samples keep improving (Fig. 3), shifting sampling times toward low context hurts far more than toward high context (μ = 0.6 vs 0.4, Fig. 5), and dropping the 1/t weight alone already gives about half the gain on LM1B (Fig. 5, middle).

    • The central equation of the method, Eq. (6), seems like established practice. Dropping the ELBO weight goes back to DDPM and has been used for masked models (Shi & Titsias, 2025; Sahoo et al., 2026). Mid-range noise levels are standard in continuous diffusion, where EDM makes the same high-noise argument (paraphrased in App. A.1), and BD3-LM (§5) applies them to MDMs with the same endpoint arguments, though the paper cites it only for block generation. Concurrent work (PUMA, InfoDLM) would also be worth discussing.

    • The theory seems to add little to the experiments: Prop. 4.2 is a known result (Zheng et al.), and the k-inefficiency of Prop. 4.4 is neither measured nor used elsewhere in the paper. Its criterion is also symmetric under t → 1−t, so it predicts the same benefit for μ = 0.4 and 0.6, whereas Fig. 5 shows a large difference.

  2. The experiments do not yet isolate the source of the gains or establish them beyond LM1B.

    • The method drops the 1/t weight and concentrates t around mid-range values at once. The effect of dropping the 1/t weight is not de-confounded outside packed LM1B, where it already accounts for about half of the gain.

    • The OWT speedup rests on one run per method, and its final gain is about half the run-to-run variation suggested by the two packed-LM1B Base runs.

    • The 774M CPT baseline scores far below the 127M DiffuGPT-S from the CPT recipe the paper adopts (Gong et al., 2025; 2.15 vs 21.6 on LAMBADA), with OpenBookQA below chance. The win rates are measured against a baseline that cannot yet perform conditional generation (App. E).

Questions for authors

Clarifications that could change my assessment:

  1. Are the two packed-LM1B Base runs (Fig. 6 vs Figs. 5 and 7–10) different seeds or different configurations? Could you explain why the headline uses the Fig. 6 run? What is the impact of seed-to-seed variance on the speedups?
  2. Which sampler produced the generations, downstream answers and instruction-following responses (decoding rule, steps, temperature, precision)? What is the sample entropy?
  3. Is the training loss a plain sum over masked tokens, as in Eq. (6), or normalized by the number of masked tokens? For the uniform run in Fig. 5, normalizing would restore roughly a 1/t weight.
  4. For CPT, what are the token count and scoring protocol? Could you comment on why the baseline is well below DiffuGPT?

Experiments that could change my assessment:

  1. OWT with at least one more training seed per method, compared with uniform sampling without the 1/t weight, BD3-LM-style clipped masking rates, and Jia et al.'s P-POTS sampler.
  2. A more extensive test of the mechanism, such as:
    • the Sec. 3 analyses repeated on OWT;
    • gradient variance for each objective;
    • a more direct locality intervention.

Suggestion for future work (does not affect my assessment):

  1. Single-epoch, larger-scale training, where dropping the weight alone saves only about 12% of compute (Sahoo et al., 2026).
4 comments
AuthorsOpenReview ↗

Author Rebuttal (1/4)

We thank the reviewer for the valuable feedback! We address the reviewer's questions below, and the changes in the revised paper are highlighted in blue.

W1. Limited evidence for locality bias and methodological novelty.

We sincerely thank the reviewer for these detailed and valuable comments, and for giving us another opportunity to clarify our contributions. For clarity, we reorganize the reviewer's concerns into three parts below. If we have misunderstood any point, we would be happy to provide further clarification.

W1.1. Is locality bias an important source of training inefficiency?

(1) Locality evidence is limited to packed LM1B.

Following the reviewer's suggestion, we extended the analyses in Secs. 3.2 and 3.3 to OpenWebText (OWT). We observed consistent findings across both datasets. Please refer to our response to Q6 and Appendix C.7 of the revised paper for the additional results.

(2) Our method does not directly address locality bias.

We would like to clarify that locality bias is the underlying factor motivating our analysis, whereas inefficient low- and high-context training samples are its consequences. Our goal is to mitigate these consequences by reducing the frequency of inefficient context regimes, rather than directly modifying the locality bias of language.

For further details, please refer to our response to W1 of Reviewer qjuJ. We also provide experiments with span masking, a more direct locality-oriented intervention, in our response to Q6.

(3) High-context samples continue improving in Fig. 3.

We would like to clarify that low- and high-context samples are training-inefficient for different reasons, as summarized in Findings 2 and 3. Specifically, low-context samples converge early and provide little additional learning signal (Finding 2), as analyzed in Sec. 3.2 and Fig. 3. In contrast, high-context samples may retain reducible error but provide redundant learning signals (Finding 3), as demonstrated by the gradient analyses in Sec. 3.3 (Tables 1 and 2). Thus, continued improvement in the high-context region in Fig. 3 does not contradict our findings. We would appreciate it if the reviewer could revisit these analyses.

(4) Shifting toward low-context regions is more harmful than shifting toward high-context regions.

As explained above, low- and high-context inefficiencies arise through different mechanisms, and therefore need not have equal effects on training. Their relative importance cannot be determined from locality bias alone, motivating our empirical comparison of different sampling distributions in Fig. 5. We believe these systematic comparisons are an important part of our contribution.

(5) Removing the 1/t1/t weighting already improves performance.

We agree and will clarify this point with additional references in the revised paper. Nevertheless, Gaussian sampling consistently outperforms uniform sampling without the 1/t1/t weighting, demonstrating benefits beyond simply removing the weighting term. Furthermore, our additional experiments in Q5 show that removing 1/t1/t alone provides little improvement on OWT, whereas our method achieves substantially better performance.

W1.2. Limited methodological novelty (EDM, BD3-LM, PUMA, ...).

We thank the reviewer for suggesting these relevant studies, which we have incorporated into the revised Related Work section (Appendix A.1).

However, we would like to clarify that our contribution is not the invention of bell-shaped time sampling itself. As already discussed in Lines 876–886 of the original submission, similar sampling strategies have been explored in continuous diffusion. Rather, our contribution is to investigate the distinct training inefficiencies of MDMs, provide an analysis motivating middle-context sampling, and empirically establish its effectiveness for MDM training.

Furthermore, in response to Q5, we conducted additional comparisons with BD3-LM, P-POTS, and uniform sampling without 1/t1/t weighting. Our method achieves the best performance among these alternatives. We would appreciate it if the reviewer could consider these additional results.

W1.3. Limited theoretical contributions.

(1) Proposition 4.2 is already established by Zheng et al.

We agree and would like to clarify that we do not claim novelty for the underlying optimizer-preservation result. Indeed, Lemma B.3 explicitly restates the result of Zheng et al.

The purpose of Proposition 4.2 is to rigorously justify our time-sampling objective. While Zheng et al. establish the result through positive weighting over context-token counts, we formulate it for general probability distributions over continuous diffusion time tt. This requires establishing that the induced context-level weights remain positive, including for distributions with atoms or endpoint mass. We provide the corresponding measure-theoretic proof in Appendix B.2 in original submission.

AuthorsOpenReview ↗

Author Rebuttal (2/4)

(2) Proposition 4.4 does not explain the asymmetric performance in Fig. 5.

We would like to clarify that Proposition 4.4 characterizes the probability of certain masking patterns, rather than predicting their precise training efficiency. In particular, it establishes that bell-shaped sampling reduces the frequency of long consecutive runs of masks or non-masks.

We agree that this symmetric criterion alone cannot explain the different performances of μ=0.4\mu=0.4 and μ=0.6\mu=0.6. However, as analyzed in Secs. 3.2 and 3.3, low- and high-context regions exhibit different sources of training inefficiency, which need not be symmetric. Therefore, the theoretical result provides a partial justification for our sampling strategy, while empirical comparisons remain necessary to determine effective sampling distributions.

We have clarified the scope of this proposition in Section 4.2 of the revised manuscript. We thank the reviewer again for the constructive feedback, which has helped us improve the presentation of our contributions.

W2. The source and robustness of the reported efficiency gains.

W2.1. The effects of removing 1/t1/t and bell-shaped sampling are not sufficiently disentangled.

We thank the reviewer for raising this concern. As discussed in W1.1(5), we have clarified this distinction in Section 4.1 of the revised paper. Importantly, Gaussian time sampling still substantially outperforms uniform sampling after removing the 1/t1/t weighting, demonstrating that the improvement is not solely attributable to removing this coefficient. This is confirmed on both LM1B and OWT, with the additional OWT results provided in Q5.

W2.2. Limited statistical evidence for the OWT speedup.

We thank the reviewer for raising this concern. We conducted three independent Gaussian runs with different random seeds to further examine the consistency of our results. The following table reports the mean and standard deviation across these runs:

Training StepsBase NLLGaussian NLL (Mean ± Std)
100K3.3003.247 ± 0.005
200K3.2333.196 ± 0.005
300K3.2003.174 ± 0.005
400K3.1793.160 ± 0.004
500K3.1673.151 ± 0.005

Gaussian time sampling consistently outperforms the Base model, with small variations across random seeds.

W2.3 & Q4. Weak CPT baseline and instruction-following comparisons.

We thank the reviewer for raising this concern. We acknowledge that the relatively weak absolute performance of our CPT models is a limitation of our experimental setting. One likely explanation is the substantially smaller training budget: our models were trained on 26B tokens, whereas DiffuGPT-S and DiffuGPT-M were trained on 131B and 209B tokens, respectively. Furthermore, differences in model size may also affect convergence, suggesting that our 774M-parameter models may not have been sufficiently trained. The relatively weak instruction-following performance may also stem from this limitation.

We acknowledge that the limited training budget is an important experimental limitation, particularly when interpreting the absolute downstream and instruction-following performance. Nevertheless, our results show that Gaussian time sampling improves performance over the standard MDM baseline under identical training conditions.

Furthermore, as discussed in our response to qjuJ (W2), we additionally compare our CPT+SFT models with their ARM initialization, GPT-2 Large. Our models achieve better performance on several benchmarks, including OBQA, TriviaQA, HellaSwag, and SIQA, suggesting that they have acquired useful capabilities despite the limited training budget.

Q1. Are the packed-LM1B results from different runs, and how does seed variance affect the reported speedups?

We sincerely thank the reviewer for carefully examining our results and identifying this discrepancy. We greatly appreciate the reviewer's attention to detail, which helped us identify an error that we had unfortunately overlooked.

Upon carefully re-examining our experimental results and plotting procedures, we confirmed that an incorrect training run was inadvertently used in Figure 6, as the reviewer suspected. We sincerely apologize for this mistake and any confusion it may have caused. We have also carefully rechecked the results presented in the other figures and confirmed that all other figures contain the correct experimental results. We have corrected Figure 6 and reported values in the revised paper and ensured that the corresponding reported results are consistent with the correct training run.

AuthorsOpenReview ↗

Author Rebuttal (3/4)

Furthermore, following the reviewer's suggestion, we additionally evaluated seed-to-seed variance using 3 different random seeds to assess the robustness of our findings. The results are presented below:

Training StepsBase NLLGaussian NLL (Mean ± Std)
100K3.6843.543 ± 0.001
200K3.5843.475 ± 0.001
300K3.5333.442 ± 0.002
400K3.5033.420 ± 0.002
500K3.4803.412 ± 0.005

We deeply regret the plotting error and are especially grateful to the reviewer for bringing it to our attention.

Q2. Which sampler produced the generations, downstream answers and instruction-following responses? What is the sample entropy?

We clarify the decoding and evaluation protocols for each experiment below, along with the corresponding sample entropy measurements. For all protocols, we added this explanation to Appendix C.1.

1. OWT Unconditional Generation.

Following the standard MDM framework, we used ancestral sampling with 512, 1,024, and 2,048 diffusion steps, temperature 1.0, and BF16 mixed precision. For each setting, we generated 100 samples across 5 random seeds and evaluated generative perplexity using GPT-2 Large in FP32.

We additionally measured the entropy of the generated samples:

Diffusion StepsBaseGaussian
5125.546 ± 0.0125.571 ± 0.012
1,0245.495 ± 0.0175.495 ± 0.007
2,0485.398 ± 0.0295.433 ± 0.006

The two methods exhibit comparable sample entropy, indicating that the generative PPL improvements are not accompanied by a substantial reduction in token diversity. We have incorporated this result in Table 9.

2. GPT-2 Large CPT/SFT Downstream Evaluation.

For multiple-choice benchmarks, following DiffuGPT, we evaluated each candidate using NELBO with averaged over 64 uniformly spaced masking rates, with independently sampled masks for each candidate. The candidate with the lowest value was selected as the answer.

3. Instruction-Following Evaluation.

For AlpacaEval 2.0 and MT-Bench, we used confidence-based categorical decoding with 256 response positions and 256 model forward passes, temperature 0.9, and FP32 inference, without top-p filtering. We evaluated 805 AlpacaEval 2.0 prompts and the first turn of 80 MT-Bench questions, using GPT-4o as the judge.

The entropies of the generated responses are:

BenchmarkBaseGaussian
AlpacaEval 2.02.2062.360
MT-Bench2.4102.551

Gaussian time sampling produces slightly higher sample entropy on both instruction-following benchmarks, suggesting that its improved win rates are not simply attributable to reduced response diversity.

Q3. Is the training loss normalized by the number of masked tokens?

We thank the reviewer for this question. As explicitly stated in the caption of Fig. 5, "Uniform" corresponds to Eq. (6) with π(t)=Uniform(0,1)\pi(t)=\mathrm{Uniform}(0,1), rather than the standard MDM NELBO. Specifically, the loss is a plain sum of cross-entropy losses over masked tokens, without normalization by the number of masked tokens. The loss coefficient is exactly 1, with no 1/t1/t weighting.

Q5. Comparison with BD3-LM and P-Pots and multiple-seed evaluation on OWT.

We thank the reviewer for this valuable suggestion. Following the reviewer's recommendation, we conducted additional experiments on OpenWebText (OWT), comparing our Gaussian time sampling with uniform sampling without 1/t1/t weighting, BD3-LM-style clipped masking rates, and Jia et al.'s P-POTS sampler. For a controlled comparison, we followed the original implementations under identical training settings, without 1/t1/t weighting for all alternative methods, while retaining the standard MDM objective (Base 1/t1/t) as a reference.

The following table reports the number of training steps required to first reach the same validation NLL achieved by the standard MDM baseline at 500K steps. The results are presented below:

MethodSteps to Target NLLSpeedup vs. Base
Base (1/t1/t)500K1.00×
Base w/o 1/t1/t380K1.32×
BD3-LM-style clipped540K0.93×
P-POTS390K1.28×
Gaussian (Ours)310K1.61×

Our Gaussian time sampling achieves faster convergence than both BD3-LM and P-POTS, further supporting its effectiveness relative to alternative training strategies.

Regarding the multiple-seed evaluation on OWT, please refer to W2.2.

AuthorsOpenReview ↗

Author Rebuttal (4/4)

Q6. A more extensive test of the mechanism

Q6.1. Sec. 3 Analyses Repeated on OWT

We thank the reviewer for this valuable suggestion! Following the reviewer's request, we repeated the analyses in Secs. 3.2 and 3.3 on OpenWebText (OWT) and observed trends consistent with LM1B. The context-level NLL analysis (Fig. 3) was also reproduced on OWT, with ARM trained for 1M steps. The results show similar trends to those in the original analysis.

Gradient cosine similarity on OWT.

Radius rr481632128
Cos. Sim.0.2920.5470.7970.8950.973

Joint-mask vs. single-mask gradient cosine similarity on OWT.

# of Masks2481632
Cos. Sim.1.0000.9990.9950.9740.953

Distant observed tokens have progressively less influence on gradients, while joint-mask gradients remain highly aligned with the sum of single-mask gradients. These findings further support the generalizability of our locality-bias analysis beyond LM1B. Detailed results, including the corresponding figure for the context-level NLL analysis (Sec. 3.2) and the gradient analyses (Sec. 3.3), are provided in Appendix C.7 of the revised paper.

Q6.2. Gradient variance for each objective

We thank the reviewer for this valuable suggestion. Following the reviewer's request, we additionally measured gradient variance across 32 mini-batches (batch size 512) at different training checkpoints on LM1B, comparing the standard MDM objective (Base 1/t1/t), uniform sampling without 1/t1/t weighting, and our Gaussian time sampling.

The results are summarized below:

Training StepsBase (1/t1/t)Uniform w/o 1/t1/tGaussian (Ours)
100K2,226.96184.83222.74
300K2,055.68210.31243.66
500K2,223.00246.32287.22
750K2,733.06304.77347.84
1M3,281.05402.51420.81

Removing the 1/t1/t weighting substantially reduces gradient variance compared with the standard MDM objective. Our Gaussian time sampling also exhibits substantially lower gradient variance than the standard baseline.

Importantly, Gaussian achieves faster convergence than uniform sampling without 1/t1/t weighting, despite exhibiting slightly higher gradient variance. This suggests that the benefits of Gaussian time sampling cannot be explained solely by variance reduction. Rather, our results support the importance of allocating more training samples to informative middle-context regions, as motivated by our analysis in Sec. 3.

Q6.3. More direct locality intervention.

We thank the reviewer for this valuable suggestion. To investigate a more direct locality-oriented masking strategy, we conducted additional experiments using SpanBERT-style masking [1], a well-established approach that masks contiguous spans rather than individual tokens independently. Specifically, we sample span lengths from Geo(0.2)\mathrm{Geo}(0.2) and repeatedly mask spans until reaching the desired masking ratio.

The test NLLs after 0.5M training steps on LM1B are:

Base MDMBase MDM + Span MaskingOursOurs + Span Masking
NLL3.485.47 (+1.99)3.415.41 (+2.00)

Span masking substantially degrades performance for both the standard MDM and our method. One possible explanation is that span masking introduces dependencies between masked positions, modifying the standard independent-masking forward process. Consequently, unlike our time-sampling strategy (Proposition 4.2), it does not come with a guarantee of preserving the theoretical optimum of the standard MDM NELBO.

Nevertheless, we do not claim that masking-ratio adjustment is necessarily the most effective approach, nor that span masking is inherently ineffective. SpanBERT was originally designed for a different training framework, and more sophisticated locality-aware masking strategies may offer further improvements. We believe this is an interesting direction for future research. The details have been added to Appendix C.8 of the revised paper.

[1] Joshi et al., "SpanBERT: Improving Pre-training by Representing and Predicting Spans," TACL'20.

S1. Single-epoch, larger-scale training, where dropping the weight alone saves only about 12% of compute (Sahoo et al., 2026).

We thank the reviewer for this thoughtful suggestion! We agree that evaluating our method under larger-scale, single-epoch training would provide valuable insights into its practical benefits, particularly in light of the findings of Sahoo et al. (2026). We believe this is an important direction for future work and appreciate the reviewer for highlighting it.

We sincerely thank the reviewer again for detailed and thoughtful review. We would be happy to address any further questions or concerns.

Reviewer_HoEHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why masked diffusion language models train more slowly than autoregressive models and proposes bell-shaped time sampling. By decomposing the objective according to the number of visible context tokens, it argues that very low-context examples are learned early while very high-context examples provide redundant gradients. The method therefore samples masking levels near t=0.5t=0.5 more often and removes the usual NELBO weight; under infinite capacity and a time-agnostic model, the population minimizer is unchanged. It reaches the baseline's final validation NLL several times faster on LM1B and OpenWebText, and reports improvements in most language-modeling and continual-pretraining evaluations.

Critical strengths
  1. A clear and practically useful reframing of "why MDMs are slow", and a method that follows from it at almost no cost. The distinction between high loss and hard to learn (Prop. 3.1: loss = irreducible conditional entropy + a learnable KL gap) is clean and correct, and the observation that the zero-context term is already at its optimum early in training is a convincing, exactly checkable data point. The resulting method is a small change to the training objective, compatible with existing MDM training recipes including CPT/SFT, and Prop. 4.2 shows that its population minimiser is unchanged under the stated assumptions. For a community that is actively trying to scale MDMs, an easy-to-adopt training-efficiency improvement of this size is significant.

  2. The empirical evidence is broad and the ablations are well chosen.

  • The speedup is consistent across LM1B (packed and unpacked) and OWT, and it also shows up in zero-shot perplexity, in downstream tasks after CPT + SFT, and in instruction-following win rates.
  • The distribution-family and hyper-parameter ablations (Gaussian / Laplace / Dirac, σ, μ) support the claim that what matters is mass near the middle rather than the specific family.
  • Sec. 5.4 is useful: the middle-flat scheduler draws more middle-context samples but compensates with NELBO weights, and importance reweighting matches the bell-shaped objective in expectation without changing which corruption levels are drawn. Neither reproduces the speedup, which suggests that both the sampled levels and the estimator matter.
Critical weaknesses
  1. The analysis and the method are only loosely connected. The method is directly motivated by avoiding very low- and high-context examples, but the paper's broader locality explanation is not tested end-to-end. The comparison between fixed and resampled random orders shows that learning many orders need not be costly, but it does not isolate locality from directionality, positional encoding, or other optimization effects. The locality claim is therefore better viewed as a plausible interpretation than an established cause of the speedup.

  2. The efficiency gain is not placed clearly in context against autoregressive models. Although the MDM–ARM training gap motivates the paper, the proposed method is compared only with the baseline MDM. The reported speedup is measured while the baseline is still improving, so it remains unclear whether final converged performance is preserved or how much of the gap to ARM is actually closed. A direct comparison or a clearer discussion would make the practical significance easier to assess.

Questions for authors

A discussion I would like to see: MDM versus ARM I would encourage the authors to add a short, balanced discussion of what the training-efficiency gap means, since it frames the whole paper:

  • Supervision per sequence. An ARM receives a loss on every token of every sequence, while an MDM only receives a loss on the masked ones. Part of the MDM–ARM gap may therefore be a plain sample-efficiency factor independent of locality.
  • Where MDMs may be better: any-order generation, infilling and bidirectional conditioning; parallel decoding with potentially fewer function evaluations.
  • Where ARMs remain better: exact likelihood and simple evaluation, cheaper training, mature KV-cache inference, and no dependency error from sampling several tokens from a product of marginals.
3 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We thank the reviewer for the valuable feedback! We address the reviewer's questions below, and the changes in the revised paper are highlighted in blue.

W1. The connection between locality bias and training inefficiency is not sufficiently isolated.

We thank the reviewer for this insightful comment. We agree that locality bias, directionality, positional encoding, and other optimization effects may be intertwined, and that a single comparison cannot fully disentangle their contributions. However, we would like to clarify two points.

First, our comparison between ARM and σ\sigma-ARM provides a controlled setting. σ\sigma-ARM uses exactly the same architecture, causal attention, and training objective as ARM. The only difference is the generation order: we consistently permute the input tokens and their corresponding RoPE positions according to a fixed permutation σ\sigma. Thus, σ\sigma-ARM is simply an ARM trained in a different fixed generation order, rather than a model with a different architecture or positional encoding mechanism. In this setting, directionality and positional information are naturally intertwined with how the generation order aligns with the local dependencies of language, which is precisely the locality-related effect we aim to investigate. Nevertheless, we acknowledge that their individual contributions cannot be completely isolated.

Second, we agree that comparing ARM with MDM introduces additional factors, and thus provide supplementary experiments. These factors can include architectural and objective differences. To better understand their relative importance, we extend the experiment in Sec. 3.1 by including MDM and systematically varying the generation order of σ\sigma-ARM.

Specifically, while the original σ\sigma-ARM uses a fully random fixed permutation, we additionally consider σr\sigma_r-ARM, which keeps the first 1−r1-r fraction of a length-LL sequence in L2R order and applies a fixed random permutation to the remaining rr fraction, for 0≤r≤10\leq r\leq1. Thus, increasing rr progressively disrupts the alignment with the original L2R structure while preserving the ARM architecture and objective.

The test NLLs after 0.5M training steps on LM1B are:

ARMσ0.25\sigma_{0.25}-ARMσ0.5\sigma_{0.5}-ARMMDMσ0.75\sigma_{0.75}-ARMσ1\sigma_{1}-ARMAoARM
3.143.343.483.483.593.673.67

We highlight two observations:

  • Within the same ARM architecture and objective, progressively disrupting the original generation order produces a substantial degradation. In particular, σ0.75\sigma_{0.75}-ARM and σ1\sigma_1-ARM exhibit NLL increases of +0.45 and +0.53 relative to ARM, respectively. These are larger than the ARM–MDM gap (+0.34). Importantly, this degradation occurs even with a single fixed generation order, without introducing the complexity of learning multiple orders.

  • MDM performs comparably to σ0.5\sigma_{0.5}-ARM and outperforms σ0.75\sigma_{0.75}-ARM, σ1\sigma_1-ARM, and AoARM. Notably, σ1\sigma_1-ARM achieves the same NLL as AoARM (3.67), despite using only a single fixed generation order. While comparisons involving MDM introduce additional architectural and objective differences, these results show that a degradation exceeding the ARM–MDM gap can already arise within the controlled ARM setting.

Taken together, these results suggest that the mismatch between generation order and the local dependency structure of language is an important contributor to slow any-order learning, with an effect comparable to or larger than the observed ARM–MDM gap. Nevertheless, this does not establish locality bias as the unique cause of slower MDM training, and we have clarified this distinction in the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

W2.1. Can you better quantify the efficiency gain relative to ARMs?

We thank the reviewer for this important suggestion. We agree that directly quantifying how much our method closes the training-efficiency gap between MDMs and ARMs would provide a clearer understanding of its practical significance.

However, the previously reported ~16× training-compute gap [1], which is cited in our paper, is based on compute-optimal scaling analyses, which estimate the Pareto-optimal frontier across different model sizes and training budgets. In contrast, our work focuses on improving MDM training efficiency under a fixed model architecture and training budget.

Consequently, directly translating our observed speedups into a reduction of the reported 16× gap would not constitute a controlled comparison. A rigorous quantification of how much the compute-optimal gap is closed would require a separate scaling study across multiple model sizes and compute budgets, which is beyond the scope of this work.

Nevertheless, our experiments demonstrate consistent improvements over standard MDM training under matched settings. We will clarify this distinction in the revised paper to better contextualize the practical significance of our results.

W2.2. Do the efficiency gains persist after substantially longer training?

To address this concern, we extended the training of both the standard MDM and our Gaussian time-sampling MDM on LM1B from 1M to 3M steps.

Our method maintains its advantage even after substantially longer training, achieving a test perplexity of 28.0 compared to 29.3 for the standard MDM at 3M steps. This demonstrates that the performance gap persists beyond the original training budget.

These results provide empirical evidence that the benefits of Gaussian time sampling extend beyond early-stage acceleration and remain effective after substantially longer training, rather than simply enabling faster convergence to the same performance level.

Q1 Please provide a balanced comparsion between autoregressive models and masked diffusion models.

We sincerely thank the reviewer for this excellent suggestion! Following the reviewer's recommendation, we have revised Section 2 to include a more balanced discussion of the strengths and limitations of MDMs and ARMs.

Specifically, we now discuss (1) differences in training efficiency and token-level supervision [1, 2], including computational overhead from masked-token processing [3], (2) the advantages of MDMs in any-order generation, infilling, and bidirectional conditioning [4], and (3) inference trade-offs, including KV-cache limitations [5, 6], the benefits of parallel decoding, and potential token-dependency errors [6]. We also provide a broader comparison of the two frameworks, drawing on additional relevant references.

We would appreciate it if the reviewer could take a look at the revised section. Thank you again for this valuable suggestion, which helped us improve the paper!

[1] Prabhudesai et al., "Diffusion Beats Autoregressive in Data-Constrained Settings," NeurIPS'25.

[2] Xue et al., "Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture," ICML'26.

[3] Deschenaux et al., "Partition Generative Modeling: Masked Modeling Without Masks," ICLR'26.

[4] Kim et al., "Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions," ICML'25.

[5] Ma et al., "dKV-Cache: The Cache for Diffusion Language Models," NeurIPS'25.

[6] Wu et al., "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding," ICLR'26.

We sincerely thank the reviewer again for detailed and thoughtful review. We would be happy to address any further questions or concerns.

Thank you for the thoughtful response and for conducting the additional experiments. They address several of my concerns and further strengthen the paper. I appreciate the authors’ effort and will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.