- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors investigated why mask diffusion models learn slower than autoregressive models during pretraining phase. They attributed the inefficiency to locality bias, and proposed a Bell-shaped sampling strategy to improve the training efficiency.
Critical strengths
- the proposal is very simple to apply, where the reported gains look decent.
- the authors did a thorough experimental and algorithmic analysis to design the sampling mechanism.
Critical weaknesses
- The proposed recipe is basically about how often different masking ratios shall be used during training, rather than actual good masking patterns. High- and low-context are essentially avoided by not having low and high masking ratios, this has less dependence with masking patterns, i.e., the whole reasoning behind locality may just boil down to token masking ratio, where patterns do not matter so much
- while the work targeted pretraining efficiency and used common metrics, the real impact can only be validated after testing in post-training and down-stream tasks. In Table 4, the authors did some testing, would be interesting to extend that, e.g., zero-shot performance, and also comparison to AR model baselines.
Questions for authors
- is masking ratio the most effective mechanism to exploit locality? besides the current masking patterns, does it make more sense to design more locality oriented patterns, e.g., random contiguous spans and etc.
- on the ordering pattern side, what is new relative the prior work Xue et al. (2025), which already compares L2R, fixed-random, and blockwise-random orders?
- Table 4 suggested the proposed sampling benefits mostly during CPT phase, very marginal during post-training and with SFT. Any explanation?
3 comments
Author Rebuttal (2/3)
(1) Zero-shot perplexity comparison with ARMs.
Although zero-shot perplexity results are already provided in Appendix C.4 (Table 8), we reproduce them below alongside the ARM baseline reported by Sahoo et al. [2] trained on same setting as ours.
| Training | PTB | Wikitext | LM1B | Lambada | AG News | Pubmed | Arxiv |
|---|---|---|---|---|---|---|---|
| ARM | 82.05 | 25.75 | 51.25 | 51.28 | 52.09 | 49.01 | 41.73 |
| Base | 115.99 | 37.76 | 72.06 | 48.18 | 68.31 | 43.15 | 37.85 |
| Gaussian (Ours) | 98.16 | 34.51 | 67.71 | 46.39 | 61.74 | 41.43 | 37.88 |
Our method improves over the base MDM on six out of seven benchmarks. Although the ARM baseline performs better on several benchmarks, our method also achieves lower perplexity on Lambada, Pubmed, and Arxiv. These comparisons provide additional context for the performance gap between MDMs and ARMs, while confirming that our method consistently improves the standard MDM baseline.
(2) Comparison with GPT-2 Large (ARM) and additional zero-shot evaluation on unseen downstream tasks.
Following the reviewer's suggestion, we additionally compare our CPT+SFT models with their autoregressive initialization, GPT-2 Large (ARM), and extend the evaluation to ARC-E and BoolQ, two benchmarks not included in our SFT training data.
We clarify that the original downstream benchmarks in Table 4 are evaluated without in-context examples (zero-shot prompting), although their training splits were included in SFT. Therefore, we additionally evaluate ARC-E and BoolQ to assess generalization to tasks unseen during SFT.
| Training | CPT | SFT | LAMBADA (%) | OBQA (%) | Wino (%) | PIQA (%) | SIQA (%) | Infilling | TriQA (%) | HellaSwag (%) | ARC-E (%) | BoolQ (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-2 Large (AR) | — | — | 40.42 | 18.00 | 50.36 | 70.78 | 37.46 | 11.05 | 4.45 | 35.68 | 46.14 | 60.95 |
| Base + Base | Base | Base | 5.82 ± 0.15 | 18.20 ± 0.28 | 50.59 ± 0.67 | 56.80 ± 0.23 | 38.95 ± 0.80 | 9.87 ± 0.15 | 6.40 ± 0.19 | 32.59 ± 1.24 | 46.49± 0.21 | 61.62± 1.33 |
| Gaussian + Gaussian | Gaussian | Gaussian | 17.89 ± 0.37 (+12.07) | 23.40 ± 1.41 (+5.20) | 49.80 ± 1.12 (-0.79) | 59.30 ± 0.88 (+2.50) | 42.07 ± 0.04 (+3.12) | 12.62 ± 0.38 (+2.75) | 27.80 ± 0.06 (+21.40) | 39.84 ± 1.21 (+7.25) | 50.88± 0.32 (+4.39) | 65.75± 1.27 (+4.13) |
The results are presented in the table below. Gaussian time sampling improves over the standard MDM baseline on most benchmarks and outperforms GPT-2 Large on several tasks, including OBQA, SIQA, TriviaQA, and HellaSwag. More importantly, on the two unseen benchmarks, Gaussian consistently outperforms both the standard MDM baseline and GPT-2 Large. These additional results suggest that the benefits of Gaussian time sampling extend beyond the benchmarks used during SFT, providing further evidence of improved downstream generalization under our training setting.
We thank the reviewer again for encouraging us to further contextualize and strengthen our empirical evaluation.
[2] Sahoo et al., "Simple and Effective Masked Diffusion Language Models," NeurIPS'24.
Author Rebuttal (3/3)
Q2. Novelty relative to Xue et al. [3] in the generation-order analysis.
We thank the reviewer for raising this point. We have already discussed the distinction from prior ordering-based studies in the Related Work section. We appreciate this opportunity to further clarify our contributions.
Xue et al. [3] demonstrate that random generation orders lead to slower convergence than L2R orders, and suggest potential explanations for this observation. Our work goes beyond this finding in two important respects.
First, we investigate whether the difficulty of any-order learning primarily arises from order-space complexity or from the mismatch between random generation orders and the locality structure of language. Specifically, Sec. 3.1 compares -ARM, which learns a single fixed random order, with AoARM, which learns variable random orders. Despite the substantially larger order space encountered by AoARM, their convergence is comparable. This suggests that learning many different orders is not necessarily the primary source of training difficulty, and that locality-related mismatch may play a more important role.
Second, and more importantly, we investigate why locality-related effects lead to inefficient MDM training, rather than merely observing that random orders slow convergence. Specifically, Sec. 3.2 shows that low-context samples quickly converge to their optimum, leaving little reducible error despite their high loss. Sec. 3.3 further demonstrates that high-context samples contain redundant learning signals, as distant observed tokens contribute little additional gradient information. These analyses identify two distinct mechanisms underlying inefficient MDM training, which are not investigated by Xue et al.
Finally, these findings directly motivate our bell-shaped time sampling strategy, which reallocates training budget away from inefficient context regimes and substantially accelerates MDM training.
In summary, while Xue et al. establish that random generation orders hinder convergence, our contribution is to further investigate why this difficulty arises, characterize its consequences for MDM training, and develop a practical strategy to mitigate them. We have further clarified the distinction from Xue et al. in Appendix A.1 (Related Work) of the revised paper.
[3] Xue et al., "Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture", ICML'26.
Q3. Why are the improvements more pronounced during CPT than SFT?
We thank the reviewer for this thoughtful question. We agree that the improvements in Table 4 primarily arise from applying Gaussian time sampling during CPT, while its additional benefits during SFT are relatively modest.
We conjecture that this difference stems from the different roles and training durations of CPT and SFT. CPT involves 100K training steps, during which the model adapts from an ARM to an MDM and learns bidirectional denoising behavior. Thus, improving training efficiency through Gaussian time sampling can have a substantial cumulative effect. In contrast, SFT involves only 10K steps and starts from an already pretrained MDM, primarily adapting its learned representations to downstream tasks rather than learning general language modeling capabilities from scratch.
Nevertheless, we acknowledge that the relative benefits may also depend on the training objective, dataset, and optimization horizon. Our results suggest that the primary practical benefit of Gaussian time sampling arises during CPT, with these improvements largely carrying over to subsequent SFT.
We sincerely thank the reviewer again for detailed and thoughtful review. We would be happy to address any further questions or concerns.
Author Rebuttal (1/3)
We thank the reviewer for the valuable feedback! We address the reviewer's questions below, and the changes in the revised paper are highlighted in blue.
W1. The proposed method adjusts masking ratios rather than directly optimizing masking patterns.
We thank the reviewer for this insightful comment. We would like to clarify that locality bias is the underlying factor motivating our analysis, whereas the inefficiency of low- and high-context samples is its consequence. Our method aims to mitigate this consequence rather than directly modify the locality bias of language.
More specifically, due to locality bias, low-context samples often lack informative nearby tokens and quickly converge to their optimum (Sec. 3.2), while high-context samples contain many distant observed tokens that contribute little additional learning signal (Sec. 3.3). This motivates reducing the frequency of these inefficient context regimes.
We agree that our method does not explicitly select individual masking patterns based on their locality. However, under the standard MDM forward process, the masking ratio directly controls the distribution of local masking patterns. In particular, Definition 4.3 and Proposition 4.4 formally establish that bell-shaped time sampling reduces the probability of long consecutive runs of masks or non-masks, which are motivated by the inefficiencies analyzed in Secs. 3.2 and 3.3.
Therefore, our contribution is not to identify optimal masking patterns individually, but to provide a simple way to reduce the occurrence of inefficient local-context configurations through time sampling.
Q1. Would more locality-oriented masking patterns, such as span masking, be more effective?
We thank the reviewer for this suggestion. We do not claim that masking-ratio adjustment is necessarily the most effective approach. However, span masking introduces consecutive masked tokens, which is contrary to our motivation of reducing long consecutive runs of masks or non-masks. Moreover, span masking modifies the standard independent-masking forward process, whereas our method preserves it.
Second, artificially adjusting masking patterns, such as span masking, might not preserve the optimum of the theoretical NELBO. For example, span masking alters the standard MDM forward process by replacing independent corruption across positions with dependent corruption, under which the standard diffusion formulation no longer applies. This contrasts with Proposition 4.2, which proves that bell-shaped time sampling preserves the same optimum as the theoretical NELBO. Thus, the optimizer-preservation guarantee in Proposition 4.2 does not directly extend to such modifications of the masking process.
To investigate this empirically, we additionally conducted experiments using the official SpanBERT-style masking strategy [1], sampling span lengths from Geo(0.2) until reaching the desired masking ratio. The test NLLs after 0.5M training steps on LM1B are:
Span masking substantially degrades performance for both the standard MDM and our method. While this does not rule out other locality-aware masking strategies, it suggests that simply introducing contiguous masked spans does not complement the training of masked diffusion models. We believe exploring more sophisticated locality-aware masking strategies is an interesting direction for future work; yet we hope reviewer will understand that we intended to provide simple and basic method that can accelerate the training of MDM. The details have been added to Appendix C.8 of the revised paper.
[1] Joshi et al., "SpanBERT: Improving Pre-training by Representing and Predicting Spans," Transactions of the association for computational linguistics, 2020.
W2. Provide broader downstream evaluation including zero-shot performance and comparison with ARMs.
We thank the reviewer for this valuable suggestion! Before presenting additional comparisons, we would like to clarify two points.
First, our paper already provides an extensive evaluation of MDM training efficiency and downstream performance. Specifically, we evaluate (1) validation NLL on three settings, (2) generative perplexity, (3) zero-shot perplexity on seven benchmarks, (4) downstream performance on eight benchmarks, and (5) instruction-following performance on two benchmarks. To our knowledge, this represents one of the more comprehensive evaluations of an MDM pretraining paper.
Second, our goal is to improve the training efficiency of existing MDMs, rather than to claim superiority over ARMs. Our primary comparisons therefore focus on standard MDM baselines. Nevertheless, we agree that comparisons with ARMs provide useful context, and we present additional results below.