- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents Phase-MoE, a joint training and inference optimization framework for MoE BDLMs, aiming to mitigate the expert explosion problem under continuous batching. The authors observe that parallel token decoding in BDLMs substantially increases the number of distinct experts activated per forward pass, while variations in denoising stages and semantic differences across concurrent requests further expand the active expert set, leading to significant HBM access overhead. To address this issue, the authors propose two main components: (1) Phase-Constrained MoE, which introduces mask-density-dependent phase biases during training to encourage phase-specific expert preferences and restricts the candidate expert set at each denoising stage to a fixed size, while preserving access to the full expert capacity during prefill; and (2) Phase-Aware Scheduler, which groups and schedules asynchronous requests according to their mask densities during inference to improve expert reuse within batches, while employing dynamic waiting thresholds to mitigate request starvation. The authors initialize and train a BDLM based on Ling-mini-2.0 and evaluate the proposed framework under continuous batching workloads. Experimental results demonstrate that Phase-MoE reduces the number of distinct activated experts by up to 56.2% and MoE kernel latency by 38.5%, while largely preserving generation quality.
Critical strengths
-
Practical relevance of the research problem: The paper addresses the expert explosion problem in MoE diffusion language models under concurrent request processing and continuous batching, rather than focusing primarily on single-request inference as in many prior studies. This setting more closely reflects real-world LLM serving scenarios, making the research practically relevant.
-
Comprehensive experimental evaluation: The paper evaluates the proposed method across different concurrency levels, execution batch sizes, expert set sizes, and scheduling strategies, with comparisons against existing methods such as DES, TEAM, and dMoE. It also provides ablation studies for the individual components, offering a relatively comprehensive assessment of the proposed framework.
Critical weaknesses
-
Insufficient justification for mask-density-based expert allocation: The proposed method partitions the denoising process into predefined phases and assigns different expert sets based on mask density. However, mask density primarily reflects the progress of denoising rather than the semantic characteristics of the tokens being processed. The paper does not provide sufficient theoretical or empirical justification for why different mask densities should correspond to distinct expert preferences. Such constraints on expert selection may limit the model's representational capacity and potentially degrade generation quality.
-
Concerns about the fairness of model comparisons and potential accuracy degradation: Phase-MoE is not directly applied to the official LLaDA2.0-mini-preview model. Instead, the authors retrain a version of LLaDA2.0-mini as their baseline and incorporate the proposed techniques into this model. However, the retrained baseline exhibits substantially lower accuracy than the official LLaDA2.0-mini model. This raises concerns about the fairness and representativeness of the evaluation, as well as whether the reported preservation of generation quality adequately reflects the performance trade-offs relative to the original model.
Questions for authors
-
Why did the authors choose to retrain a new baseline model rather than directly integrating Phase-MoE into the official LLaDA2.0-mini-preview? Given the substantial accuracy degradation observed in the retrained checkpoint compared to the official model, could the authors provide evaluation results demonstrating the effectiveness of Phase-MoE when applied to the official LLaDA2.0-mini-preview
-
The IFEval accuracy reported for Phase-MoE in Figure 5 is substantially higher than that in Table 1, which is further supported by the results in Table 5. Could the authors explain why simply changing the serving scheduling strategy from FCFA to PA leads to such a significant improvement in generation accuracy, particularly on IFEval, while similar improvements are not observed on other benchmarks?
-
Could the authors provide further theoretical justification or empirical evidence demonstrating that mask-density-based expert allocation has minimal impact on model accuracy? Since mask density does not directly reflect token semantics, it is unclear why expert routing should be constrained according to denoising phases. Could such constraints effectively be interpreted as pruning a subset of candidate experts, potentially excluding semantically relevant ones? If so, to what extent can the proposed Soft Shaping mechanism compensate for the resulting loss of representational capacity?
4 comments
3. Response to Question 2: Explaining IFEval Differences (Table 1 vs. Table 5, and FCFS vs. PA)
- Table 1 vs. Table 5 / Figure 5c (Full Suite vs. Stratified Subsample): Table 1 evaluates full standalone benchmarks (all 541 IFEval prompts). To evaluate continuous batching across an exhaustive concurrency/batch grid tractably, Table 5 / Figure 5c uses a fixed stratified subsample across all six tasks (Appendix E.1), slightly shifting baseline means across all models (e.g., Top-128 scores 49.3% on full IFEval in Table 1 vs. 49.0%–52.0% in Table 5).
- FCFS vs. PA Within Table 5 (Batch-Composition & BF16 Numerical Sensitivity): PA scheduling does not alter the per-sequence decoding algorithm. However, in continuous batching, each forward step pads the active sequences to the batch's maximum cached context length and executes attention/projections in
bfloat16. Because FCFS and PA group different sequences together, the resulting batch tensor shapes and padding masks differ, which can introduce minor floating-point non-associativity differences. When a candidate token's confidence lies on the hard unmasking boundary (), minor numerical differences can flip whether that token is unmasked at step or , cascading into trajectory differences on strict binary formatting checks like IFEval. Crucially, Table 5 shows these shifts occur in both directions rather than systematically favoring PA: switching at lowers Baseline IFEval (), slightly raises Top-128 (), and on Top-96 () raises IFEval () while lowering GSM8K (). We will document this sensitivity in Appendix E.1.
We are grateful for your constructive feedback and hope these new results and clarifications address your concerns. Please let us know if any questions remain; if our response resolves the issues raised, we kindly hope you will consider updating your evaluation accordingly.
Official Comment by Reviewer J2Bm
Thank you for your response and the additional experiments. I remain concerned about the applicability of the method to more thoroughly trained models, as manually grouping experts could degrade the performance of frontier MoE models. I will therefore maintain my original rating.
We thank Reviewer J2Bm for engaging with our rebuttal. To ensure the technical record is clear regarding how Phase-MoE operates and where it fits into the frontier training lifecycle, we briefly clarify three points:
-
Data-Driven Self-Organization, Not Manual Grouping: Phase-MoE does not manually group or heuristically assign experts to denoising phases. At the start of Stage 1, the phase-bias matrix is initialized to zero (), leaving all experts equally accessible across all phases. As prior diffusion literature observes—and as our Normalized Mutual Information trajectory (Figure 6) confirms—diffusion routers naturally develop temporal preferences as mask density shifts from coarse planning () to local syntactic refinement (). Stage 1 Soft Shaping (Equation 3) simply amplifies this organic, data-driven self-organization into smooth diagonal phase bands (Figure 3) while expert FFNs co-adapt via gradient descent. In fact, we agree that manually or statically grouping experts degrades performance: that is precisely what our
Direct Hard Maskablation (Appendix G; Table 12) tests, where imposing static phase groups without Soft Shaping causes severe routing thrashing (phase_max_vio) and a drop on GSM8K that our Soft-to-Hard curriculum explicitly resolves. -
Target Entry Point in the Frontier Lifecycle (AR BDLM Transition vs. Post-Hoc Retrofitting): Phase-MoE is designed to be introduced when a model first enters the temporal diffusion environment—whether during from-scratch diffusion pre-training or during Autoregressive-to-BDLM Continual Pre-Training (CPT). We evaluate the AR BDLM route starting from Ling-mini-2.0 () because it is the standard paradigm adopted by frontier BDLMs (LLaDA 2.0, DiffusionGemma, SDAR), and provides a clean, strictly controlled A/B testbed before experts form unconstrained cross-phase dependencies. Starting from
Ling-mini-2.0is therefore not a proxy workaround; it is the point before experts have formed unconstrained cross-phase dependencies. Conversely, applying phase constraints post-hoc onto an already finished BDLM (llada2.0-mini-preview) whose experts have already settled into an unconstrained equilibrium over diffusion tokens would force already-converged diffusion experts to re-specialize after the fact, running counter to our training-time co-design philosophy; retrofitting phase constraints post-hoc would require a different adaptation mechanism, which we leave to future work. -
Scaling to Longer AR BDLM Training Horizons ( Tokens): When applied at this AR-to-BDLM inception point, scaling the training budget from our controlled -token A/B testbed to a frontier -token CPT pipeline works in favor of Phase-MoE. Once Stage 2 locks the candidate pools (preserving all experts in Phase 0 and semantic combinations per phase in
Top-128), the remainder of CPT and SFT consists of standard gradient updates where expert FFNs adapt to their active phase bands. A compact -token horizon is the stringent regime because FFNs have fewer post-pruning steps to co-adapt; longer training horizons provide an order of magnitude more gradient updates to absorb the phase boundary and either narrow the remaining quality gap further or sustain tighter budgets for the same HBM bandwidth and throughput gains.
We appreciate your time and feedback throughout the review process.
We thank Reviewer J2Bm for highlighting the practical relevance of solving the expert explosion under realistic continuous batching and our comprehensive experimental evaluation. Below, we address your concerns and questions, referencing Global Rebuttal Sections 1.2 & 1.4 and Tables R1–R3.
1. Response to Weakness 1 & Question 3: Mask-Density Allocation and Soft-Shaping Compensation
As detailed in Global Rebuttal §1.2, conditioning candidate pools on mask density ϕ is grounded in diffusion information dynamics and verified by our curriculum ablations (Table 9):
[MASK]embeddings (greater than 0.85 cosine similarity) for coarse planning, whereas late denoising (ϕ→1) refines local syntax over revealed context (Wu et al., 2026)—an affinity learned organically from Mϕ,j=0 (Figure 6 NMI). Semantics are preserved via Phase 0 Isolation (100% of 4,864 experts active during unconstrained prefill/commit, Table R3) and Intra-Phase Semantic Routing ((8128)≈1.4×1012 semantic k=8 combinations within Khard=128, with 4,864/4,864 experts active in denoising, Table R3).phase_max_vio(1.14→7.68) and collapsing GSM8K from 58.8% to 30.5% (Top-128) and 25.0% (Top-96). Gradually shaping M>0 before hard pruning allows router biases and expert FFNs to co-adapt into their active phase bands—recovering +24.8 pp on GSM8K for Top-128 (30.5%→55.3%, recovering 87.6% of the static-mask loss) and +12.6 pp on MMLU (50.5%→63.1%), staying within 1.3 to 3.5 pp of the unconstrained baseline on 7 of 8 standalone benchmarks (and within 1.0 pp on IFEval under continuous batching, Table R2) while cutting expert fetches by 44.9%.2. Response to Weakness 2 & Question 1: Controlled 12B-Token A/B Testbed vs. Official LLaDA2.0-mini-preview
We did not evaluate Soft-to-Hard continual adaptation starting from LLaDA2.0-mini-preview; instead, we branched both our Baseline (B=32) and Phase-MoE variants from Ling-mini-2.0 (the exact AR base checkpoint used by LLaDA 2.0) for two experimental design reasons:
phase_max_vio=7.68).