- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper is about training a large MoE diffusion language model based on the LLaDa architecture. The authors conduct controlled small-scale experiments to establish scaling laws with respect to hyperparameters, different compute allocations between data and parameters, and architecture details, such as expert granularity and activation sparsity. Guided by this study, they design a 30B-A3B architecture and training scheme, and train the model on around 23.5T tokens. They then evaluate the model on a suite of reasoning and coding benchmarks, and compare it to some recent DLLMs and a MoE LLM of the same size. The results indicate strong performance among the presented DLLMs.
Critical strengths
- The small-scale experiments are set up well and cover many modeling choices along the dimensions of hyperparameters, compute allocation and MoE architecture, the authors clearly explain what relationships they are studying and select the experiments accordingly.
- The large-scale model testing is set up to follow the findings to a reasonable degree, and the trained model scores well on the demonstrated benchmarks.
Critical weaknesses
Although the small-scale scaling experiments are systematic and reasonably well-controlled, the large-scale comparisons do not clearly isolate the contribution of the proposed scaling laws to the final model's performance. The models used for comparison differ significantly from LLaDA MoE v2 in architecture, optimization, training objectives, data, and/or model size. While these comparisons demonstrate that LLaDA MoE v2 is a competitive diffusion language model, they do not establish to what extent its performance improvements arise from the scaling-law-guided design rather than other differences in the training setup. In particular, Figure 5 compares LLaDA MoE v2 with the previous LLaDA MoE, but the models differ substantially in size and training configuration. A controlled comparison of the proposed recipe against conventional AR-derived optimization and architecture choices, under matched training conditions at an intermediate scale, would provide stronger evidence for the practical value of the proposed scaling laws.
Questions for authors
- If we train the same diffusion model with the same budget, does following these scaling laws improve performance compared with using conventional AR-derived hyperparameters and MoE design choices? The current study focuses on comparisons within the scaling setup of the paper.
- How sensitive are these laws to the timestep sampling? The scaling laws assume a uniform timestep sampling distribution, which is not the only choice. To which extent would these laws apply to other timestep sampling strategies?
- How robust are the final model evaluations to the decoding settings? The models used in the main table use different decoding procedures and settings.
- How was the activation ratio of 9.09% chosen? The sweep seems to suggest that lower values are even better.
5 comments
Response to Reviewer 2Ht7 (2/3)
Q2: Applicability to other timestep schedules
Prior work shows that noise schedules of different functional forms yield the same likelihood objective in expectation, yet result in different estimator variances (Table 9 in [1]). Among them, the uniform masking schedule achieves the lowest empirical variance in [1] and has demonstrated consistent overall advantages across prior masked dLLM studies [1–4]. Given this, the uniform schedule we use is precisely this lowest-variance, empirically validated choice, making it best suited for scaling up.
Nevertheless, it remains valuable to examine to what extent our fitted laws apply to other timestep sampling strategies. While repeating all scaling sweeps across schedules is infeasible within the rebuttal window, we focus on optimization hyperparameters as the component most directly governed by these estimator variance differences. Following the search protocol in Section 3.1, we run the same joint grid search over at FLOPs under two non-uniform schedules [1, 2]: Poly2 (, skewing toward low mask rates) and Cosine (, skewing toward high mask rates), and compare our frozen law predictions against each schedule's empirical grid optimum.
As shown in Table A, our uniform-calibrated hyperparameter laws apply well to both non-uniform schedules. Without refitting, our laws match the grid-optimal batch size and predict an adjacent learning rate . Although this predicted setting is not the exact grid optimum, under both schedules it falls within the near-optimal region defined in Appendix A.3, confirming that our scaling laws remain viable beyond uniform sampling.
Motivated by the reviewer's question, we have added these experimental details and grid heatmaps to Appendix A.7 of the revised paper and highlighted them in blue.
Table A: Full hyperparameter search grids (training loss) at FLOPs. Bold marks the configuration predicted by our scaling laws, and † marks empirical near-optimal configurations (within 0.25% of each schedule's minimum loss). Under both schedules, our prediction (bold†) falls within the empirical near-optimal region.
| Schedule | Batch Size | ||||
|---|---|---|---|---|---|
| Poly2 | 4.069 | 4.012 | 3.978 | 3.979 | |
| Poly2 | 3.958 | 3.894 | 3.903 | 3.877 | |
| Poly2 | 3.842 | 3.820 | 3.828 | 3.827 | |
| Poly2 | 3.809† | 3.808† | 3.800† | 3.815 | |
| Cosine | 4.178 | 4.134 | 4.103 | 4.075 | |
| Cosine | 4.048 | 4.023 | 3.963 | 3.954 | |
| Cosine | 3.961 | 3.928 | 3.901 | 3.900 | |
| Cosine | 3.914 | 3.886† | 3.880† | 3.881† |
[1] Sahoo et al., Simple and Effective Masked Diffusion Language Models, https://arxiv.org/abs/2406.07524.
[2] Shi et al., Simplified and Generalized Masked Diffusion for Discrete Data, https://arxiv.org/abs/2406.04329.
[3] Nie et al., Large Language Diffusion Models, https://arxiv.org/abs/2502.09992.
[4] Ni et al., Training Optimal Large Diffusion Language Models, https://arxiv.org/abs/2510.03280.
Response to Reviewer 2Ht7 (3/3)
Q3: Robustness to decoding settings
In the original comparison, we used block size 64 for LLaDA MoE v2 to match the LLaDA MoE evaluation protocol [1] and kept it fixed across benchmarks to avoid task-specific tuning. For a fair comparison, we used published baseline scores when available. Otherwise, we evaluated each baseline using the decoding settings recommended in its original paper, including a block size of 4 for SDAR [2].
To test the robustness of this comparison, we evaluated LLaDA MoE v2 Base and Instruct on representative conditional generation benchmarks at several block sizes. As shown in Tables A and B, the overall lead of both LLaDA MoE v2 variants over SDAR is robust across the tested block sizes, including SDAR's recommended setting of 4. These results corroborate the overall performance advantage reported in the paper.
Following the reviewer's question, we have added these results to Appendix B.4 of the revised paper and highlighted them in blue.
Table A: Conditional generation results for LLaDA MoE v2 Base.
| Model | KorBench | GSM8K | HumanEval |
|---|---|---|---|
| SDAR Sci Base | 40.08 | 86.13 | 33.54 |
| LLaDA MoE v2 Base (Block = 4) | 45.52 | 88.32 | 48.17 |
| LLaDA MoE v2 Base (Block = 16) | 45.76 | 88.17 | 50.00 |
| LLaDA MoE v2 Base (Block = 64) | 45.92 | 83.93 | 50.00 |
| LLaDA MoE v2 Base (Block = 256) | 46.32 | 88.25 | 47.56 |
Table B: Conditional generation results for LLaDA MoE v2 Instruct.
| Model | MATH | Olympiad | MBPP | LCBv6 |
|---|---|---|---|---|
| SDAR Chat | 77.80 | 34.93 | 71.60 | 21.70 |
| LLaDA MoE v2 Instruct (Block = 4) | 79.26 | 47.56 | 81.03 | 26.10 |
| LLaDA MoE v2 Instruct (Block = 64) | 80.02 | 46.44 | 81.03 | 27.75 |
| LLaDA MoE v2 Instruct (Block = 256) | 78.56 | 44.41 | 81.50 | 27.20 |
[1] Zhu et al., LLaDA-MoE: A Sparse MoE Diffusion Language Model, https://arxiv.org/abs/2509.24389.
[2] Cheng et al., SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation, https://arxiv.org/abs/2510.06303.
Q4: Choice of activation ratio
As the reviewer notes, in terms of training loss alone, Figure 4(a) shows a monotonic decrease as we add routed experts and reduce the activation ratio. The sweep therefore favors lower activation ratios and leaves the training-loss optimum undetermined. By contrast, the sweeps over expert granularity and the shared-expert ratio identify a near-optimal range and a consistent optimum, respectively.
In practice, however, a larger expert pool must be spread over more GPUs under expert parallelism, so a larger share of token-to-expert transfers crosses GPUs. The original parallel decoding algorithm for dLLMs does not support KV caching, so each denoising step recomputes the prefix's KV states and routes all tokens through the MoE layers again. As a result, these cross-GPU token transfers repeat at every decoding step and can add substantial communication overhead. We therefore limited the expert pool to 128 routed experts to balance training gains against inference communication costs, yielding an activation ratio of 9.09% in our final design.
In response to the reviewer's question, we have added this discussion to Section 4 of the revised paper and highlighted it in blue.
We thank Reviewer 2Ht7 again for the constructive review and welcome any further questions or requests for clarification.
Response to Reviewer 2Ht7 (1/3)
We thank Reviewer 2Ht7 for the insightful feedback and for recognizing our systematic experimental study, clear presentation, and strong benchmark performance at scale. We respond below to the weakness (W) and questions (Q) raised in the review.
W1 & Q1: Isolating scaling-law contributions
When training the same MoE dLLM with the same budget under matched conditions, following our scaling laws consistently improves over conventional AR-derived hyperparameters, allocation rules, and MoE design choices. Following the reviewer's suggestion, we run this controlled intermediate-scale comparison within our scaling sweeps, which share the same corpus, dLLM objective, and implementation. Replacing one component at a time with prior AR or dLLM reference settings [1–7] at matched budgets while keeping all other choices fixed, we report relative loss reduction at the same budget and efficiency gain (EG), a standard metric in scaling studies [1, 8, 9]: if our setting and a reference reach the same loss at training FLOPs Cours and Cref, EG=Cref/Cours, so an EG of 1.1 means the reference needs 10% more FLOPs and EG>1 favors our setting.
Table A shows clearly where our efficiency gains come from. The largest leverage lies in optimization hyperparameters, where prior AR laws require 50%–75% more FLOPs to match our loss because they under-predict the large batch sizes favored by masked diffusion. Beyond optimization, our allocation law outperforms both Ling's AR MoE law and the dense dLLM law of DLMs across budgets, showing that MoE sparsity and the diffusion objective interact and must be calibrated jointly. For architecture, evaluating LLaDA MoE's AR-inherited design under matched model size and training setup resolves the confounding factors in Figure 5, yielding consistent gains across every dimension, while the only entries below parity against Ling and Qwen3 reflect our deliberate cap on activation sparsity for multi-step decoding efficiency, as detailed in Q4. Taken together, each part of our recipe addresses a distinct mismatch between conventional AR heuristics and MoE dLLM training dynamics.
Although component-wise EGs cannot simply be multiplied to decompose our 30B model's benchmark gains, these controlled intermediate-scale results confirm that our scaling-law-guided design brings genuine improvements beyond differences in model size, data, or training setup.
In light of the reviewer's feedback, we have added these results to Section 3.4 and Appendix A.6 of the revised paper and highlighted them in blue.
Table A: Component-wise efficiency gains. Entries are reported as our relative loss reduction / EG (EG > 1 favors ours). – indicates that the original paper does not report the corresponding scaling study or that its setting does not match our compute budgets.
[1] Tian et al., Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models, https://arxiv.org/abs/2507.17702.
[2] Bi et al., DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, https://arxiv.org/abs/2401.02954.
[3] Dai et al., DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, https://arxiv.org/abs/2401.06066.
[4] Ling Team, Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation, https://arxiv.org/abs/2510.22115.
[5] Yang et al., Qwen3 Technical Report, https://arxiv.org/abs/2505.09388.
[6] Zhu et al., LLaDA-MoE: A Sparse MoE Diffusion Language Model, https://arxiv.org/abs/2509.24389.
[7] von Rütte et al., Scaling Behavior of Discrete Diffusion Language Models, https://arxiv.org/abs/2512.10858.
[8] Microsoft AI Team, Mai-thinking-1: Building a hill-climbing machine, https://microsoft.ai/pdf/mai-thinking-1.pdf.
[9] Liu et al., Muon is Scalable for LLM Training, https://arxiv.org/abs/2502.16982.