acceptodds
Under review as a conference paper at ICLR 2027

Beyond Autoregressive Recipes: Scaling Up Mixture-of-Experts Diffusion LLMs to 30B

Abstract

Diffusion large language models (dLLMs) are a promising alternative to autoregressive (AR) models, but existing sparse Mixture-of-Experts (MoE) dLLMs largely inherit AR recipes, and the masked denoising objective calls for dLLM-specific scaling laws. We present a systematic study of scaling laws for MoE dLLMs, spanning optimization hyperparameters, compute allocation, and MoE architecture across a broad range of compute budgets. Guided by these laws, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens; it surpasses the previous work LLaDA MoE at only half of the latter's training FLOPs. The base model approaches the strong open-weight AR MoE model Qwen3 30B-A3B on knowledge, reasoning, and coding benchmarks while using \(65%\) as many pretraining tokens. After supervised fine-tuning, it outperforms SDAR Chat, a competitive MoE dLLM, on seven of eight reasoning and coding benchmarks. Together, these results show LLaDA MoE v2 to be the strongest dLLM trained from scratch to date, establishing dLLM-specific scaling laws as a foundation for large-scale MoE dLLM training and practical application.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Scaling Law, Foundation Models, Diffusion Large Language Models, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2Ht7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is about training a large MoE diffusion language model based on the LLaDa architecture. The authors conduct controlled small-scale experiments to establish scaling laws with respect to hyperparameters, different compute allocations between data and parameters, and architecture details, such as expert granularity and activation sparsity. Guided by this study, they design a 30B-A3B architecture and training scheme, and train the model on around 23.5T tokens. They then evaluate the model on a suite of reasoning and coding benchmarks, and compare it to some recent DLLMs and a MoE LLM of the same size. The results indicate strong performance among the presented DLLMs.

Critical strengths
  1. The small-scale experiments are set up well and cover many modeling choices along the dimensions of hyperparameters, compute allocation and MoE architecture, the authors clearly explain what relationships they are studying and select the experiments accordingly.
  2. The large-scale model testing is set up to follow the findings to a reasonable degree, and the trained model scores well on the demonstrated benchmarks.
Critical weaknesses

Although the small-scale scaling experiments are systematic and reasonably well-controlled, the large-scale comparisons do not clearly isolate the contribution of the proposed scaling laws to the final model's performance. The models used for comparison differ significantly from LLaDA MoE v2 in architecture, optimization, training objectives, data, and/or model size. While these comparisons demonstrate that LLaDA MoE v2 is a competitive diffusion language model, they do not establish to what extent its performance improvements arise from the scaling-law-guided design rather than other differences in the training setup. In particular, Figure 5 compares LLaDA MoE v2 with the previous LLaDA MoE, but the models differ substantially in size and training configuration. A controlled comparison of the proposed recipe against conventional AR-derived optimization and architecture choices, under matched training conditions at an intermediate scale, would provide stronger evidence for the practical value of the proposed scaling laws.

Questions for authors
  1. If we train the same diffusion model with the same budget, does following these scaling laws improve performance compared with using conventional AR-derived hyperparameters and MoE design choices? The current study focuses on comparisons within the scaling setup of the paper.
  2. How sensitive are these laws to the timestep sampling? The scaling laws assume a uniform timestep sampling distribution, which is not the only choice. To which extent would these laws apply to other timestep sampling strategies?
  3. How robust are the final model evaluations to the decoding settings? The models used in the main table use different decoding procedures and settings.
  4. How was the activation ratio of 9.09% chosen? The sweep seems to suggest that lower values are even better.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2Ht7 (1/3)

We thank Reviewer 2Ht7 for the insightful feedback and for recognizing our systematic experimental study, clear presentation, and strong benchmark performance at scale. We respond below to the weakness (W) and questions (Q) raised in the review.

W1 & Q1: Isolating scaling-law contributions

When training the same MoE dLLM with the same budget under matched conditions, following our scaling laws consistently improves over conventional AR-derived hyperparameters, allocation rules, and MoE design choices. Following the reviewer's suggestion, we run this controlled intermediate-scale comparison within our scaling sweeps, which share the same corpus, dLLM objective, and implementation. Replacing one component at a time with prior AR or dLLM reference settings [1–7] at matched budgets while keeping all other choices fixed, we report relative loss reduction at the same budget and efficiency gain (EG), a standard metric in scaling studies [1, 8, 9]: if our setting and a reference reach the same loss at training FLOPs CoursC_{\mathrm{ours}} and CrefC_{\mathrm{ref}}, EG=Cref/Cours\mathrm{EG}=C_{\mathrm{ref}}/C_{\mathrm{ours}}, so an EG of 1.1 means the reference needs 10% more FLOPs and EG>1\mathrm{EG}>1 favors our setting.

Table A shows clearly where our efficiency gains come from. The largest leverage lies in optimization hyperparameters, where prior AR laws require 50%–75% more FLOPs to match our loss because they under-predict the large batch sizes favored by masked diffusion. Beyond optimization, our allocation law outperforms both Ling's AR MoE law and the dense dLLM law of DLMs across budgets, showing that MoE sparsity and the diffusion objective interact and must be calibrated jointly. For architecture, evaluating LLaDA MoE's AR-inherited design under matched model size and training setup resolves the confounding factors in Figure 5, yielding consistent gains across every dimension, while the only entries below parity against Ling and Qwen3 reflect our deliberate cap on activation sparsity for multi-step decoding efficiency, as detailed in Q4. Taken together, each part of our recipe addresses a distinct mismatch between conventional AR heuristics and MoE dLLM training dynamics.

Although component-wise EGs cannot simply be multiplied to decompose our 30B model's benchmark gains, these controlled intermediate-scale results confirm that our scaling-law-guided design brings genuine improvements beyond differences in model size, data, or training setup.

In light of the reviewer's feedback, we have added these results to Section 3.4 and Appendix A.6 of the revised paper and highlighted them in blue.

Table A: Component-wise efficiency gains. Entries are reported as our relative loss reduction / EG (EG > 1 favors ours). – indicates that the original paper does not report the corresponding scaling study or that its setting does not match our compute budgets.

ComponentFLOPsDeepSeek LLM [2]DeepSeekMoE [3]Ling [1, 4]Qwen3 [5]LLaDA MoE [6]DLMs [7]
Hyperparameters2e194.15% / 1.75–1.03% / 1.50–––
Allocation3e18––1.07% / 1.17––1.50% / 1.23
Allocation3e19––0.33% / 1.06––1.18% / 1.24
Activation6e19–0.48% / 1.08−1.47% / 0.79−0.73% / 0.890.54% / 1.09–
Granularity6e19–1.12% / 1.170.00% / 1.000.57% / 1.080.70% / 1.10–
Shared experts6e19–0.12% / 1.020.21% / 1.030.62% / 1.090.62% / 1.09–

[1] Tian et al., Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models, https://arxiv.org/abs/2507.17702.

[2] Bi et al., DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, https://arxiv.org/abs/2401.02954.

[3] Dai et al., DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, https://arxiv.org/abs/2401.06066.

[4] Ling Team, Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation, https://arxiv.org/abs/2510.22115.

[5] Yang et al., Qwen3 Technical Report, https://arxiv.org/abs/2505.09388.

[6] Zhu et al., LLaDA-MoE: A Sparse MoE Diffusion Language Model, https://arxiv.org/abs/2509.24389.

[7] von Rütte et al., Scaling Behavior of Discrete Diffusion Language Models, https://arxiv.org/abs/2512.10858.

[8] Microsoft AI Team, Mai-thinking-1: Building a hill-climbing machine, https://microsoft.ai/pdf/mai-thinking-1.pdf.

[9] Liu et al., Muon is Scalable for LLM Training, https://arxiv.org/abs/2502.16982.

AuthorsOpenReview ↗

Response to Reviewer 2Ht7 (2/3)

Q2: Applicability to other timestep schedules

Prior work shows that noise schedules of different functional forms yield the same likelihood objective in expectation, yet result in different estimator variances (Table 9 in [1]). Among them, the uniform masking schedule achieves the lowest empirical variance in [1] and has demonstrated consistent overall advantages across prior masked dLLM studies [1–4]. Given this, the uniform schedule we use is precisely this lowest-variance, empirically validated choice, making it best suited for scaling up.

Nevertheless, it remains valuable to examine to what extent our fitted laws apply to other timestep sampling strategies. While repeating all scaling sweeps across schedules is infeasible within the rebuttal window, we focus on optimization hyperparameters as the component most directly governed by these estimator variance differences. Following the search protocol in Section 3.1, we run the same 4×44\times 4 joint grid search over (B,η)(B, \eta) at 2×10182\times 10^{18} FLOPs under two non-uniform schedules [1, 2]: Poly2 (m(t)=t2m(t)=t^2, skewing toward low mask rates) and Cosine (m(t)=sin⁡(πt/2)m(t)=\sin(\pi t/2), skewing toward high mask rates), and compare our frozen law predictions against each schedule's empirical grid optimum.

As shown in Table A, our uniform-calibrated hyperparameter laws apply well to both non-uniform schedules. Without refitting, our laws match the grid-optimal batch size 2202^{20} and predict an adjacent learning rate 2.0×10−32.0\times 10^{-3}. Although this predicted setting is not the exact grid optimum, under both schedules it falls within the near-optimal region defined in Appendix A.3, confirming that our scaling laws remain viable beyond uniform sampling.

Motivated by the reviewer's question, we have added these experimental details and grid heatmaps to Appendix A.7 of the revised paper and highlighted them in blue.

Table A: Full 4×44\times 4 hyperparameter search grids (training loss) at 2×10182\times 10^{18} FLOPs. Bold marks the configuration predicted by our scaling laws, and † marks empirical near-optimal configurations (within 0.25% of each schedule's minimum loss). Under both schedules, our prediction (bold†) falls within the empirical near-optimal region.

ScheduleBatch Size BBη=2.8×10−3\eta = 2.8\times 10^{-3}η=2.0×10−3\eta = 2.0\times 10^{-3}η=1.4×10−3\eta = 1.4\times 10^{-3}η=1.0×10−3\eta = 1.0\times 10^{-3}
Poly22172^{17}4.0694.0123.9783.979
Poly22182^{18}3.9583.8943.9033.877
Poly22192^{19}3.8423.8203.8283.827
Poly22202^{20}3.809†3.808†3.800†3.815
Cosine2172^{17}4.1784.1344.1034.075
Cosine2182^{18}4.0484.0233.9633.954
Cosine2192^{19}3.9613.9283.9013.900
Cosine2202^{20}3.9143.886†3.880†3.881†

[1] Sahoo et al., Simple and Effective Masked Diffusion Language Models, https://arxiv.org/abs/2406.07524.

[2] Shi et al., Simplified and Generalized Masked Diffusion for Discrete Data, https://arxiv.org/abs/2406.04329.

[3] Nie et al., Large Language Diffusion Models, https://arxiv.org/abs/2502.09992.

[4] Ni et al., Training Optimal Large Diffusion Language Models, https://arxiv.org/abs/2510.03280.

AuthorsOpenReview ↗

Response to Reviewer 2Ht7 (3/3)

Q3: Robustness to decoding settings

In the original comparison, we used block size 64 for LLaDA MoE v2 to match the LLaDA MoE evaluation protocol [1] and kept it fixed across benchmarks to avoid task-specific tuning. For a fair comparison, we used published baseline scores when available. Otherwise, we evaluated each baseline using the decoding settings recommended in its original paper, including a block size of 4 for SDAR [2].

To test the robustness of this comparison, we evaluated LLaDA MoE v2 Base and Instruct on representative conditional generation benchmarks at several block sizes. As shown in Tables A and B, the overall lead of both LLaDA MoE v2 variants over SDAR is robust across the tested block sizes, including SDAR's recommended setting of 4. These results corroborate the overall performance advantage reported in the paper.

Following the reviewer's question, we have added these results to Appendix B.4 of the revised paper and highlighted them in blue.

Table A: Conditional generation results for LLaDA MoE v2 Base.

ModelKorBenchGSM8KHumanEval
SDAR Sci Base40.0886.1333.54
LLaDA MoE v2 Base (Block = 4)45.5288.3248.17
LLaDA MoE v2 Base (Block = 16)45.7688.1750.00
LLaDA MoE v2 Base (Block = 64)45.9283.9350.00
LLaDA MoE v2 Base (Block = 256)46.3288.2547.56

Table B: Conditional generation results for LLaDA MoE v2 Instruct.

ModelMATHOlympiadMBPPLCBv6
SDAR Chat77.8034.9371.6021.70
LLaDA MoE v2 Instruct (Block = 4)79.2647.5681.0326.10
LLaDA MoE v2 Instruct (Block = 64)80.0246.4481.0327.75
LLaDA MoE v2 Instruct (Block = 256)78.5644.4181.5027.20

[1] Zhu et al., LLaDA-MoE: A Sparse MoE Diffusion Language Model, https://arxiv.org/abs/2509.24389.

[2] Cheng et al., SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation, https://arxiv.org/abs/2510.06303.

Q4: Choice of activation ratio

As the reviewer notes, in terms of training loss alone, Figure 4(a) shows a monotonic decrease as we add routed experts and reduce the activation ratio. The sweep therefore favors lower activation ratios and leaves the training-loss optimum undetermined. By contrast, the sweeps over expert granularity and the shared-expert ratio identify a near-optimal range and a consistent optimum, respectively.

In practice, however, a larger expert pool must be spread over more GPUs under expert parallelism, so a larger share of token-to-expert transfers crosses GPUs. The original parallel decoding algorithm for dLLMs does not support KV caching, so each denoising step recomputes the prefix's KV states and routes all tokens through the MoE layers again. As a result, these cross-GPU token transfers repeat at every decoding step and can add substantial communication overhead. We therefore limited the expert pool to 128 routed experts to balance training gains against inference communication costs, yielding an activation ratio of 9.09% in our final design.

In response to the reviewer's question, we have added this discussion to Section 4 of the revised paper and highlighted it in blue.

We thank Reviewer 2Ht7 again for the constructive review and welcome any further questions or requests for clarification.

Reviewer_PKRrOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This papers presents scaling laws for MoE dLLMs, and trains them based on these insights from scaling laws. They show great empirical gains and are able to beat most baselines on a wide variety of tasks, thus empirically validating their results.

Critical strengths
  1. The paper studies scaling laws for the training MoE models for dLLMs. This research is empirical but offers practical insights for the community about how to pre-train dLLMs from scratch.
  2. Their proposed dLLM is much better on average than other baselines, and almost gives AR-like performance.
  3. The paper is easy to read and understand, with an interesting departure from AR scaling laws.
Critical weaknesses

My main weakness/complaint with this paper is that in practice, very few people pre-train dLLMs from scratch, and most dllms out there are obtained from existing ARMs. I believe if there was an effort done to establish scaling laws for continued pre-training, it would have been much more useful. Then this paper could have claimed that they have produced scaling laws for both pre-training and continued pre-training.

Given the fact that scaling laws are entirely empirical, doing so would make the paper much stronger.

Questions for authors
  1. Can the authors try and present some scaling laws for continued pre-training of MoE dlms?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PKRr

We thank Reviewer PKRr for the positive feedback and for recognizing our study's practical insights, strong performance across diverse benchmarks, and interesting departures from AR scaling. We respond below to the weakness (W) and question (Q) raised in the review.

W1 & Q1: Continued-pretraining scaling laws

We agree that continued pretraining (CPT) from existing AR models is an effective, compute-efficient route to obtaining capable dLLMs, and that studying scaling laws for CPT is valuable. At the same time, pretraining from scratch plays an important complementary role by exploring the unconstrained potential of dLLMs and uncovering dLLM-specific scaling laws across optimization, compute allocation, and architecture to guide training. Regarding CPT scaling laws, unlike from-scratch laws that depend solely on training compute, CPT scaling depends jointly on the base checkpoint's scale, pretraining history, and continuation budget, making full multi-scale CPT laws infeasible within the rebuttal window. To provide initial empirical evidence, we initialize an MoE dLLM from OLMoE-1B-7B [1] and follow Section 3.1 to run a 4×54\times 5 joint search over batch size BB and learning rate η\eta at 3×10183\times 10^{18} FLOPs.

As shown in Table A, our from-scratch hyperparameter laws may not be directly suitable for guiding CPT. Specifically, at the grid-optimal learning rate, switching to our predicted batch size raises loss only slightly from 3.223 to 3.251, whereas using our predicted learning rate degrades loss sharply to 7.349. CPT requires a much smaller learning rate than from-scratch pretraining to preserve the capabilities stored from AR pretraining, consistent with prior studies adapting dLLMs from AR models such as Dream 7B [2]. Effective CPT recipes may also require checkpoint-specific analysis across different AR base models.

More broadly, CPT and from-scratch pretraining offer complementary strengths. While CPT efficiently reuses pretrained AR representations under modest continuation budgets, it naturally inherits the base checkpoint's fixed architecture: for instance, OLMoE [1] and Qwen3 [3] omit shared experts, whereas our sweeps show that MoE dLLMs consistently favor a 33.3% shared-expert ratio. Pretraining from scratch complements CPT by enabling compute allocation and MoE architecture to be jointly optimized across a broader design space. Guided by our from-scratch recipe, LLaDA MoE v2 outperforms the scale-matched AR-CPT model SDAR [4] across Base and SFT evaluations using 37% fewer total tokens while approaching strong AR counterparts. Meanwhile, systematically establishing scaling laws for CPT remains a promising direction for future work.

Following the reviewer's suggestion, we have added these CPT results and discussion to the Conclusion section and Appendix A.9 of the revised paper and highlighted them in blue.

Table A: dLLM training loss across the CPT hyperparameter search grid on OLMoE. Bold marks the configuration predicted by our from-scratch scaling laws, and † marks empirical near-optimal configurations within 0.25% of the grid minimum.

Batch Size BBη=4×10−3\eta = 4\times 10^{-3}η=2×10−3\eta = 2\times 10^{-3}η=1×10−3\eta = 1\times 10^{-3}η=5×10−4\eta = 5\times 10^{-4}η=1×10−4\eta = 1\times 10^{-4}
2182^{18}5.3515.3413.9933.7433.241
2192^{19}6.4556.3424.0093.6513.223†
2202^{20}7.6047.3493.9553.5763.251
2212^{21}7.6347.6304.0423.5333.343

[1] Muennighoff et al., OLMoE: Open Mixture-of-Experts Language Models, https://arxiv.org/abs/2409.02060.

[2] Ye et al., Dream 7B: Diffusion Large Language Models, https://arxiv.org/abs/2508.15487.

[3] Yang et al., Qwen3 Technical Report, https://arxiv.org/abs/2505.09388.

[4] Cheng et al., SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation, https://arxiv.org/abs/2510.06303.

We thank Reviewer PKRr again for the thoughtful review and welcome any further questions or requests for clarification.

Thank you for clarifying my doubts, I have decided to keep my score.

We are glad that our additional analysis clarified your questions, and we sincerely appreciate your feedback in helping improve our paper.

Reviewer_KZjkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the scaling behavior of MoE diffusion language models, covering optimization hyperparameters, model–data allocation, and expert architecture. The findings guide the training of LLaDA MoE v2, a 30B-A3B model pretrained from scratch on 23.5T tokens. The model achieves competitive results across knowledge, reasoning, and coding tasks. The main contributions are practical training insights and the release of capable diffusion language models for further research.

Critical strengths
  1. The released Base and Instruct models are valuable resources for the community. They provide a useful starting point for research on diffusion decoding, post-training, and MoE architectures, without requiring researchers to pretrain their own models.

  2. The study covers several important training and architectural choices across multiple compute budgets. These experiments provide practical guidance in an area where training recipes are still developing.

  3. The resulting model shows competitive downstream performance. The combination of systematic experiments and a successful larger-scale model makes the work useful beyond the individual scaling fits.

Critical weaknesses
  1. The methodological novelty is mostly empirical. Some claims about diffusion-specific scaling behavior are stronger than the evidence supports, since comparisons with AR scaling laws involve different data and training setups.
  2. The reliability of the fitted laws could be better established. Uncertainty estimates and sensitivity analyses are limited, particularly for conclusions about slightly different scaling exponents and consistently optimal architectural choices.
  3. The larger-scale results support the effectiveness of the overall recipe, but do not clearly isolate the contribution of the scaling laws from model size, architecture, and data choices.
Questions for authors
  1. How sensitive are the fitted exponents and preferred architectural settings to the fitting range and training variability?
  2. Is there evidence from existing experiments that helps separate the benefits of the proposed recipe from changes in model size or data?
  3. Could the authors provide more details about the data composition, benchmark decontamination, and evaluation settings?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KZjk (1/4)

We thank Reviewer KZjk for the constructive feedback and for recognizing our work's practical value, the systematic scaling experiments, and competitive performance across benchmarks. We respond below to the weaknesses (W) and questions (Q) raised in the review.

W1: Diffusion-specific scaling behavior claims

We agree that the observed differences from published AR scaling laws cannot be attributed solely to the choice between masked diffusion and AR objectives, because training data and frameworks differ across studies. However, a fully matched comparison would require repeating the experiments reported in prior studies, which is computationally prohibitive. Prior studies have also compared scaling laws fitted under their own experimental settings with published results. For example, Quokka [1] compares its fitted scaling coefficients with the Chinchilla [2] coefficients in Table 1, while DLMs [3] compares its scaling results with published AR laws in Table 1 and Figure 2. Such cross-study comparisons are common in scaling-law research and provide useful context for interpreting scaling trends and guiding large-scale model training.

In our study, controlled sweeps under a unified corpus and implementation reveal shifts from published AR laws that align qualitatively with masked denoising mechanics: for example, our 3.4×3.4\times larger optimal batch size than DeepSeek LLM [4] at 102010^{20} FLOPs is consistent with masked diffusion supervising only half of the tokens in expectation. Moreover, seeing consistent shifts across multiple independent AR baselines points to genuine properties of MoE dLLM training rather than artifacts of any data or infrastructure setup. Still, since cross-study comparisons cannot fully isolate the modeling objective, we view them as empirical intuition rather than strict evidence.

In light of the reviewer's comment, we have made this limitation explicit in the Conclusion section of the revised paper and highlighted it in blue.

[1] Ni et al., Training Optimal Large Diffusion Language Models, https://arxiv.org/abs/2510.03280.

[2] Hoffmann et al., Training Compute-Optimal Large Language Models, https://arxiv.org/abs/2203.15556.

[3] von Rütte et al., Scaling Behavior of Discrete Diffusion Language Models, https://arxiv.org/abs/2512.10858.

[4] Bi et al., DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, https://arxiv.org/abs/2401.02954.

AuthorsOpenReview ↗

Response to Reviewer KZjk (2/4)

W2 & Q1: Scaling-law reliability and sensitivity

Following common scaling-law practices [1, 2, 3], we incorporated several safeguards into our experimental protocol to ensure reliable fits. First, to suppress step-to-step stochasticity and avoid overfitting to noisy point minima in flat loss basins, we average each run's loss over the final phase of training and fit our hyperparameter laws across near-optimal configurations rather than isolated grid minima. Second, our scaling studies are conducted systematically across multiple model scales and two to three orders of magnitude in compute under a unified corpus, objective, and implementation. Third, for hyperparameter scaling, we directly validate out-of-range extrapolation at twice the largest fitting budget in Figure 2.

To directly address Q1 on the sensitivity of our fitted exponents, we design two targeted checks matching the two sources of uncertainty: sub-range refits test sensitivity to the fitting range by dropping one compute budget at a time and refitting on the remaining budgets, while noise perturbation tests sensitivity to training variability by adding Gaussian noise calibrated to our sweep residuals onto each run's loss and repeating selection and regression across Monte Carlo trials. As summarized in Table A, all fitted exponents remain concentrated within narrow bands under both checks. This directly supports our conclusions on subtle exponent differences: our faster learning-rate decay relative to prior AR laws [2, 3] and slight data-side allocation tilt relative to dense dLLM laws [4, 5] hold consistently across fitting ranges and loss noise. Meanwhile, the batch-size exponent stays close to prior AR laws, consistent with our main finding that B∗B^* requires a much larger overall batch size.

Regarding our preferred architectural settings, because Section 3.3 derives empirical design principles rather than fitting parametric curves over compute, exponent-based refits do not apply; instead, our search protocol directly addresses both fitting-range sensitivity and training variability. Specifically, replicating independent sweeps across five compute budgets spanning two orders of magnitude directly verifies that our optimal architectural choices hold consistently across scales, while isolating one dimension at a time under matched activated FLOPs in a realistic overtraining regime and conservatively reporting a near-optimal interval rather than a single value in flat basins, as in expert granularity, prevent run-to-run variability from skewing our conclusions.

Taken together, our cross-scale experimental protocol grounded in established scaling practices, the stability confirmed by our range and noise sensitivity analyses, and the successful scaling of LLaDA MoE v2 up to 30B with strong downstream performance jointly demonstrate the reliability and practical effectiveness of our scaling laws.

Following the reviewer's suggestion, we have added these analyses to Appendix A.8 of the revised paper and highlighted them in blue.

Table A: Sensitivity of fitted scaling exponents to the fitting range and training variability. Sub-range refits test fitting-range sensitivity across single-budget exclusions, and noise perturbation tests training-variability sensitivity via the central interval across Monte Carlo trials.

Scaling LawFitted ExponentSub-Range RefitsNoise Perturbation
Batch size B∗B^*0.34810.3481[0.298,0.391][0.298, 0.391][0.306,0.353][0.306, 0.353]
Learning rate η∗\eta^*−0.2447-0.2447[−0.263,−0.222][-0.263, -0.222][−0.323,−0.222][-0.323, -0.222]
Model allocation M∗M^*0.4750.475[0.460,0.479][0.460, 0.479][0.466,0.488][0.466, 0.488]
Data allocation D∗D^*0.5250.525[0.521,0.540][0.521, 0.540][0.512,0.535][0.512, 0.535]

[1] Hoffmann et al., Training Compute-Optimal Large Language Models, https://arxiv.org/abs/2203.15556.

[2] Bi et al., DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, https://arxiv.org/abs/2401.02954.

[3] Tian et al., Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models, https://arxiv.org/abs/2507.17702.

[4] von Rütte et al., Scaling Behavior of Discrete Diffusion Language Models, https://arxiv.org/abs/2512.10858.

[5] Ni et al., Training Optimal Large Diffusion Language Models, https://arxiv.org/abs/2510.03280.

AuthorsOpenReview ↗

Response to Reviewer KZjk (3/4)

W3 & Q2: Isolating contributions of scaling laws

Our existing scaling sweeps provide controlled evidence that disentangles the contribution of our scaling laws from model size, architecture, and data choices. Specifically, to isolate each component of our scaling recipe and separate its benefits from changes in model size or data, we evaluate reference settings from prior AR and dLLM studies [1–7] directly within these sweeps, which share the same corpus, dLLM objective, and implementation. Replacing one component at a time at matched training budgets, we measure efficiency gain (EG), a standard metric in scaling studies [1, 8, 9]: if our setting and a reference reach the same loss with training FLOPs CoursC_{\mathrm{ours}} and CrefC_{\mathrm{ref}}, then EG=Cref/Cours\mathrm{EG}=C_{\mathrm{ref}}/C_{\mathrm{ours}}, so an EG of 1.1 means the reference needs 10% more FLOPs and EG>1\mathrm{EG}>1 favors our setting.

Table A directly answers Q2: with the corpus, compute budget, and all other model, architecture, and training choices held fixed in each sweep, following our scaling laws consistently improves compute efficiency over prior designs. For hyperparameters, our laws achieve EGs of 1.50–1.75 over the AR laws of Ling and DeepSeek LLM. For compute allocation, our law achieves EGs of 1.06–1.24 over Ling's AR MoE law and the dense dLLM law of DLMs, showing that the gain comes from our predicted model–data split rather than extra data or compute. For MoE architecture, our design improves every dimension over LLaDA MoE's AR-inherited architecture (EGs of 1.09–1.10) and outperforms most AR MoE designs.

Although component-wise EGs cannot simply be multiplied to decompose the exact gains of our 30B model, these results confirm that our recipe itself improves efficiency independently of changes in model size or data.

To address the reviewer's feedback, we have added these results to Section 3.4 and Appendix A.6 of the revised paper and highlighted them in blue.

Table A: Component-wise efficiency gains. Entries are reported as our relative loss reduction / EG (EG > 1 favors ours). – indicates that the original paper does not report the corresponding scaling study or that its setting does not match our compute budgets.

ComponentFLOPsDeepSeek LLM [2]DeepSeekMoE [3]Ling [1, 4]Qwen3 [5]LLaDA MoE [6]DLMs [7]
Hyperparameters2e194.15% / 1.75–1.03% / 1.50–––
Allocation3e18––1.07% / 1.17––1.50% / 1.23
Allocation3e19––0.33% / 1.06––1.18% / 1.24
Activation6e19–0.48% / 1.08−1.47% / 0.79−0.73% / 0.890.54% / 1.09–
Granularity6e19–1.12% / 1.170.00% / 1.000.57% / 1.080.70% / 1.10–
Shared experts6e19–0.12% / 1.020.21% / 1.030.62% / 1.090.62% / 1.09–

[1] Tian et al., Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models, https://arxiv.org/abs/2507.17702.

[2] Bi et al., DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, https://arxiv.org/abs/2401.02954.

[3] Dai et al., DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, https://arxiv.org/abs/2401.06066.

[4] Ling Team, Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation, https://arxiv.org/abs/2510.22115.

[5] Yang et al., Qwen3 Technical Report, https://arxiv.org/abs/2505.09388.

[6] Zhu et al., LLaDA-MoE: A Sparse MoE Diffusion Language Model, https://arxiv.org/abs/2509.24389.

[7] von Rütte et al., Scaling Behavior of Discrete Diffusion Language Models, https://arxiv.org/abs/2512.10858.

[8] Microsoft AI Team, Mai-thinking-1: Building a hill-climbing machine, https://microsoft.ai/pdf/mai-thinking-1.pdf.

[9] Liu et al., Muon is Scalable for LLM Training, https://arxiv.org/abs/2502.16982.

AuthorsOpenReview ↗

Response to Reviewer KZjk (4/4)

Q3: Data, benchmark decontamination, and evaluation details

We provide the requested details on data composition, benchmark decontamination, and evaluation settings below.

Data composition. Our 23.5T-token pretraining corpus combines general knowledge text from web pages, books, encyclopedias, and academic papers with reasoning-oriented code and mathematics corpora covering source code, technical documentation, mathematical literature, problem sets, and synthetic data. While primarily English, the mixture includes approximately 4% multilingual text across roughly 30 languages, curated via rule-based and model-based cleaning, quality filtering, deduplication, and harmful-content removal. Across our five-stage schedule, Stages 1 and 2 each train on 10T tokens while raising the share of mathematics and code data from 32% to 46% and reducing general text from 68% to 54%; Stage 3 anneals for 2T tokens on a curated 1T-token high-quality subset; and Stages 4 and 5 blend long-form documents and extended reasoning sequences up to 32K tokens over 500B and 1T tokens, respectively. For SFT, we fine-tune on 7M instruction–response pairs focused primarily on mathematical reasoning and code generation under the same cleaning and deduplication pipeline.

Benchmark decontamination. Across the pretraining corpus, we screen documents against the test splits of all downstream evaluation benchmarks and remove documents that match or overlap benchmark prompts or reference solutions, applying stricter filtering to the reasoning-heavy Stage 2 mixture and the Stage 3 annealing corpus. Crucially, our 7M SFT dataset undergoes the exact same benchmark decontamination procedure to exclude any overlapping prompts or responses prior to fine-tuning.

Evaluation settings. For base-model evaluations, multiple-choice tasks use conditional likelihood following the LLaDA and LLaDA MoE protocols, while all remaining base-model tasks and all SFT benchmarks use conditional generation; when official baseline scores are unavailable, models are evaluated under a unified pipeline using each model's native likelihood or generation procedure. Both Base and Instruct conditional-generation tasks share the same decoding configuration: we allow up to 1,024 generated tokens by default and 4,096 tokens on MATH, OlympiadBench, AIME 2024, and AIME 2025, using semi-autoregressive sampling at temperature 0 with low-confidence remasking and denoising steps equal to the generation length. For ease of use and to avoid task-specific tuning, we apply a uniform block size of 64 across all main-text generative evaluations, and additionally include block-size ablations in Appendix B.4 confirming consistent advantages across block sizes.

In response to the reviewer's request, we have added these details to Appendices B.2–B.4 of the revised paper and highlighted them in blue.

We thank Reviewer KZjk again for the careful evaluation of our work and welcome any further questions or requests for clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and ACs

We sincerely thank all reviewers for their valuable comments and the AC, SAC, and PC for organizing this review process.

To summarize, we establish from-scratch scaling laws for Mixture-of-Experts diffusion language models across hyperparameters, compute allocation, and architecture, guiding the pretraining of LLaDA MoE v2 30B-A3B to state-of-the-art performance among from-scratch diffusion language models.

We are encouraged that all reviewers found our paper clear and easy to follow, valued our systematic scaling experiments and practical pretraining guidance, and recognized the strong downstream performance of LLaDA MoE v2. Reviewers further highlighted our departure from autoregressive scaling laws and near-autoregressive performance (Reviewer PKRr), our well-controlled experimental setup (Reviewer 2Ht7), and the community value of our LLaDA MoE v2 Base and Instruct models (Reviewer KZjk).

Specifically, we addressed the reviewers' comments as follows, with all manuscript revisions highlighted in blue:

  • Isolating scaling-law contributions: Controlled comparisons confirm that each component of our recipe improves compute efficiency independently of model size or data (Reviewers KZjk, 2Ht7).
  • Reliability and sensitivity: Fitting-range and training-variability sensitivity analyses confirm that our fitted exponents and architectural choices remain robust (Reviewer KZjk).
  • Cross-study comparisons: We clarified the scope of cross-study comparisons with autoregressive laws (Reviewer KZjk).
  • Data, decontamination, and evaluation details: We added detailed descriptions of our data composition, pretraining and SFT benchmark decontamination, and unified evaluation configurations (Reviewer KZjk).
  • Continued pretraining: We explored transferring our scaling laws to continued pretraining and noted systematically establishing continued-pretraining scaling laws as future work (Reviewer PKRr).
  • Timestep sampling schedules: Additional experiments under non-uniform schedules confirm that our hyperparameter laws remain near-optimal without refitting (Reviewer 2Ht7).
  • Decoding settings and activation ratio: Additional evaluations confirm consistent gains across decoding settings, and we clarified the inference-efficiency trade-off behind our activation ratio (Reviewer 2Ht7).

We thank all reviewers again for their constructive feedback, which has significantly improved this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.