acceptodds
Under review as a conference paper at ICLR 2027

Phase-MoE: A Co-Design to Bound the Expert Explosion in Block Diffusion Language Models

Abstract

Block Diffusion Language Models (BDLMs) have emerged as a compelling paradigm bridging autoregressive modeling and discrete diffusion, pairing inter-block causal generation with intra-block bidirectional parallel decoding. To scale these architectures efficiently, integrating Mixture-of-Experts (MoE) is essential. However, combining MoE with block-parallel decoding triggers a systemic Expert Explosion: processing multiple tokens simultaneously causes their independently routed experts to span the expert pool. This inflates memory traffic from High-Bandwidth Memory (HBM) to SRAM, creating a memory-bound bottleneck. We demonstrate that existing post-hoc mitigations break down under production continuous batching, where the temporal and semantic heterogeneity of asynchronous requests escalates global memory traffic to near-dense levels. To resolve this, we propose a bandwidth-aware co-design comprising Phase-Constrained MoE and a Phase-Aware Scheduler. Recognizing that discrete diffusion progresses through predictable mask-density phases, we condition the router's expert pools on the temporal denoising phase during training, leaving compute-bound prefill unconstrained. The Phase-Aware Scheduler subsequently clusters asynchronous requests by mask density at runtime, promoting overlapping expert execution. Under production-scale continuous batching simulations where existing mitigations fail, our co-design reduces unique expert activations by up to 56.2% and cuts MoE kernel latency by up to 38.5%, preserving competitive generative performance.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, Discrete Diffusion Models, Block Diffusion Language Models, Continuous Batching, Efficient Inference, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_J2BmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Phase-MoE, a joint training and inference optimization framework for MoE BDLMs, aiming to mitigate the expert explosion problem under continuous batching. The authors observe that parallel token decoding in BDLMs substantially increases the number of distinct experts activated per forward pass, while variations in denoising stages and semantic differences across concurrent requests further expand the active expert set, leading to significant HBM access overhead. To address this issue, the authors propose two main components: (1) Phase-Constrained MoE, which introduces mask-density-dependent phase biases during training to encourage phase-specific expert preferences and restricts the candidate expert set at each denoising stage to a fixed size, while preserving access to the full expert capacity during prefill; and (2) Phase-Aware Scheduler, which groups and schedules asynchronous requests according to their mask densities during inference to improve expert reuse within batches, while employing dynamic waiting thresholds to mitigate request starvation. The authors initialize and train a BDLM based on Ling-mini-2.0 and evaluate the proposed framework under continuous batching workloads. Experimental results demonstrate that Phase-MoE reduces the number of distinct activated experts by up to 56.2% and MoE kernel latency by 38.5%, while largely preserving generation quality.

Critical strengths
  1. Practical relevance of the research problem: The paper addresses the expert explosion problem in MoE diffusion language models under concurrent request processing and continuous batching, rather than focusing primarily on single-request inference as in many prior studies. This setting more closely reflects real-world LLM serving scenarios, making the research practically relevant.

  2. Comprehensive experimental evaluation: The paper evaluates the proposed method across different concurrency levels, execution batch sizes, expert set sizes, and scheduling strategies, with comparisons against existing methods such as DES, TEAM, and dMoE. It also provides ablation studies for the individual components, offering a relatively comprehensive assessment of the proposed framework.

Critical weaknesses
  1. Insufficient justification for mask-density-based expert allocation: The proposed method partitions the denoising process into predefined phases and assigns different expert sets based on mask density. However, mask density primarily reflects the progress of denoising rather than the semantic characteristics of the tokens being processed. The paper does not provide sufficient theoretical or empirical justification for why different mask densities should correspond to distinct expert preferences. Such constraints on expert selection may limit the model's representational capacity and potentially degrade generation quality.

  2. Concerns about the fairness of model comparisons and potential accuracy degradation: Phase-MoE is not directly applied to the official LLaDA2.0-mini-preview model. Instead, the authors retrain a version of LLaDA2.0-mini as their baseline and incorporate the proposed techniques into this model. However, the retrained baseline exhibits substantially lower accuracy than the official LLaDA2.0-mini model. This raises concerns about the fairness and representativeness of the evaluation, as well as whether the reported preservation of generation quality adequately reflects the performance trade-offs relative to the original model.

Questions for authors
  1. Why did the authors choose to retrain a new baseline model rather than directly integrating Phase-MoE into the official LLaDA2.0-mini-preview? Given the substantial accuracy degradation observed in the retrained checkpoint compared to the official model, could the authors provide evaluation results demonstrating the effectiveness of Phase-MoE when applied to the official LLaDA2.0-mini-preview

  2. The IFEval accuracy reported for Phase-MoE in Figure 5 is substantially higher than that in Table 1, which is further supported by the results in Table 5. Could the authors explain why simply changing the serving scheduling strategy from FCFA to PA leads to such a significant improvement in generation accuracy, particularly on IFEval, while similar improvements are not observed on other benchmarks?

  3. Could the authors provide further theoretical justification or empirical evidence demonstrating that mask-density-based expert allocation has minimal impact on model accuracy? Since mask density does not directly reflect token semantics, it is unclear why expert routing should be constrained according to denoising phases. Could such constraints effectively be interpreted as pruning a subset of candidate experts, potentially excluding semantically relevant ones? If so, to what extent can the proposed Soft Shaping mechanism compensate for the resulting loss of representational capacity?

4 comments

We thank Reviewer J2Bm for highlighting the practical relevance of solving the expert explosion under realistic continuous batching and our comprehensive experimental evaluation. Below, we address your concerns and questions, referencing Global Rebuttal Sections 1.2 & 1.4 and Tables R1–R3.

1. Response to Weakness 1 & Question 3: Mask-Density Allocation and Soft-Shaping Compensation

As detailed in Global Rebuttal §1.2, conditioning candidate pools on mask density ϕ\phi is grounded in diffusion information dynamics and verified by our curriculum ablations (Table 9):

  • Information Regimes & 2D Hierarchy: Early denoising (ϕ→B\phi \to B) processes uniform [MASK] embeddings (greater than 0.850.85 cosine similarity) for coarse planning, whereas late denoising (ϕ→1\phi \to 1) refines local syntax over revealed context (Wu et al., 2026)—an affinity learned organically from Mϕ,j=0M_{\phi,j}=0 (Figure 6 NMI). Semantics are preserved via Phase 0 Isolation (100% of 4,864 experts active during unconstrained prefill/commit, Table R3) and Intra-Phase Semantic Routing ((1288)≈1.4×1012\binom{128}{8} \approx 1.4 \times 10^{12} semantic k=8k=8 combinations within Khard=128K_{\mathrm{hard}}=128, with 4,864/4,864 experts active in denoising, Table R3).
  • Quantifying Soft-Shaping Compensation (Table 9): Imposing fixed phase supports of size KhardK_{\mathrm{hard}} from the start of training without Soft Shaping (Direct Hard Mask, Table 9 & Appendix G.5) prematurely locks tokens out of pre-trained semantic experts, spiking phase_max_vio (1.14→7.681.14 \to 7.68) and collapsing GSM8K from 58.8% to 30.5% (Top-128) and 25.0% (Top-96). Gradually shaping M>0M_{\gt 0} before hard pruning allows router biases and expert FFNs to co-adapt into their active phase bands—recovering +24.8 pp on GSM8K for Top-128 (30.5%→55.3%30.5\% \to 55.3\%, recovering 87.6% of the static-mask loss) and +12.6 pp on MMLU (50.5%→63.1%50.5\% \to 63.1\%), staying within 1.3 to 3.5 pp of the unconstrained baseline on 7 of 8 standalone benchmarks (and within 1.0 pp on IFEval under continuous batching, Table R2) while cutting expert fetches by 44.9%.

2. Response to Weakness 2 & Question 1: Controlled 12B-Token A/B Testbed vs. Official LLaDA2.0-mini-preview

We did not evaluate Soft-to-Hard continual adaptation starting from LLaDA2.0-mini-preview; instead, we branched both our Baseline (B=32B=32) and Phase-MoE variants from Ling-mini-2.0 (the exact AR base checkpoint used by LLaDA 2.0) for two experimental design reasons:

  1. Why Post-Hoc Masking Alone Is Insufficient: Phase-MoE is a training-time co-design rather than an inference-only heuristic. As shown by our Direct Hard Mask ablation (Table 9), imposing static phase supports without Stage 1 Soft Shaping causes severe routing thrashing (phase_max_vio =7.68= 7.68).
  2. Isolating Routing Effects from Closed Post-Training Pipelines: While LLaDA2.0-mini-preview branches from the same Ling-mini-2.0 base, it is the product of an unreleased post-training pipeline comprising >100B>100\text{B} proprietary CPT tokens, offline Top-kk checkpoint merging, 1.5M-pair DPO/RL alignment, and Confidence-Aware Parallel (CAP) losses (Bie et al., 2025; Appendix C.1). Running 12B-token continual pre-training on top of an already DPO/CAP-aligned checkpoint risks confounding post-training distribution shift with the structural effect of phase-constrained routing. Branching both Baseline (B=32B=32) and Phase-MoE from Ling-mini-2.0 under the exact same 12B-token open curriculum (10B SmolLM-Corpus CPT + 2B OpenHermes-2.5 SFT) ensures a simple, reproducible A/B comparison. We will clarify in Table 1's caption that llada2.0-mini-preview is shown as an external frontier reference and note continual adaptation of post-trained checkpoints as future work.

3. Response to Question 2: Explaining IFEval Differences (Table 1 vs. Table 5, and FCFS vs. PA)

  • Table 1 vs. Table 5 / Figure 5c (Full Suite vs. Stratified Subsample): Table 1 evaluates full standalone benchmarks (all 541 IFEval prompts). To evaluate continuous batching across an exhaustive concurrency/batch grid tractably, Table 5 / Figure 5c uses a fixed stratified subsample across all six tasks (Appendix E.1), slightly shifting baseline means across all models (e.g., Top-128 scores 49.3% on full IFEval in Table 1 vs. 49.0%–52.0% in Table 5).
  • FCFS vs. PA Within Table 5 (Batch-Composition & BF16 Numerical Sensitivity): PA scheduling does not alter the per-sequence decoding algorithm. However, in continuous batching, each forward step pads the BstepB_{\mathrm{step}} active sequences to the batch's maximum cached context length and executes attention/projections in bfloat16. Because FCFS and PA group different sequences together, the resulting batch tensor shapes and padding masks differ, which can introduce minor floating-point non-associativity differences. When a candidate token's confidence lies on the hard unmasking boundary (τconf=0.97\tau_{\mathrm{conf}} = 0.97), minor numerical differences can flip whether that token is unmasked at step tt or t+1t+1, cascading into trajectory differences on strict binary formatting checks like IFEval. Crucially, Table 5 shows these shifts occur in both directions rather than systematically favoring PA: switching FCFS→PA\text{FCFS} \to \text{PA} at Bstep=16B_{\mathrm{step}}=16 lowers Baseline IFEval (54.33%→52.00%54.33\% \to 52.00\%), slightly raises Top-128 (49.00%→51.00%49.00\% \to 51.00\%), and on Top-96 (Bstep=8B_{\mathrm{step}}=8) raises IFEval (47.67%→50.33%47.67\% \to 50.33\%) while lowering GSM8K (64.00%→56.00%64.00\% \to 56.00\%). We will document this sensitivity in Appendix E.1.

We are grateful for your constructive feedback and hope these new results and clarifications address your concerns. Please let us know if any questions remain; if our response resolves the issues raised, we kindly hope you will consider updating your evaluation accordingly.

Reviewer_J2BmOpenReview ↗

Official Comment by Reviewer J2Bm

Thank you for your response and the additional experiments. I remain concerned about the applicability of the method to more thoroughly trained models, as manually grouping experts could degrade the performance of frontier MoE models. I will therefore maintain my original rating.

We thank Reviewer J2Bm for engaging with our rebuttal. To ensure the technical record is clear regarding how Phase-MoE operates and where it fits into the frontier training lifecycle, we briefly clarify three points:

  1. Data-Driven Self-Organization, Not Manual Grouping: Phase-MoE does not manually group or heuristically assign experts to denoising phases. At the start of Stage 1, the phase-bias matrix is initialized to zero (M>0=0M_{>0} = \mathbf{0}), leaving all N=256N=256 experts equally accessible across all phases. As prior diffusion literature observes—and as our Normalized Mutual Information trajectory (Figure 6) confirms—diffusion routers naturally develop temporal preferences as mask density shifts from coarse planning (ϕ→B\phi \to B) to local syntactic refinement (ϕ→1\phi \to 1). Stage 1 Soft Shaping (Equation 3) simply amplifies this organic, data-driven self-organization into smooth diagonal phase bands (Figure 3) while expert FFNs co-adapt via gradient descent. In fact, we agree that manually or statically grouping experts degrades performance: that is precisely what our Direct Hard Mask ablation (Appendix G; Table 12) tests, where imposing static phase groups without Soft Shaping causes severe routing thrashing (phase_max_vio =7.68= 7.68) and a 24.8–27.4 pp24.8\text{--}27.4\text{ pp} drop on GSM8K that our Soft-to-Hard curriculum explicitly resolves.

  2. Target Entry Point in the Frontier Lifecycle (AR →\to BDLM Transition vs. Post-Hoc Retrofitting): Phase-MoE is designed to be introduced when a model first enters the temporal diffusion environment—whether during from-scratch diffusion pre-training or during Autoregressive-to-BDLM Continual Pre-Training (CPT). We evaluate the AR →\to BDLM route starting from Ling-mini-2.0 (B=1B=1) because it is the standard paradigm adopted by frontier BDLMs (LLaDA 2.0, DiffusionGemma, SDAR), and provides a clean, strictly controlled A/B testbed before experts form unconstrained cross-phase dependencies. Starting from Ling-mini-2.0 is therefore not a proxy workaround; it is the point before experts have formed unconstrained cross-phase dependencies. Conversely, applying phase constraints post-hoc onto an already finished BDLM (llada2.0-mini-preview) whose experts have already settled into an unconstrained equilibrium over >100B>100\text{B} diffusion tokens would force already-converged diffusion experts to re-specialize after the fact, running counter to our training-time co-design philosophy; retrofitting phase constraints post-hoc would require a different adaptation mechanism, which we leave to future work.

  3. Scaling to Longer AR →\to BDLM Training Horizons (>100B>100\text{B} Tokens): When applied at this AR-to-BDLM inception point, scaling the training budget from our controlled 12B12\text{B}-token A/B testbed to a frontier >100B>100\text{B}-token CPT pipeline works in favor of Phase-MoE. Once Stage 2 locks the candidate pools Cϕ\mathcal{C}_\phi (preserving all 256256 experts in Phase 0 and (1288)≈1.4×1012\binom{128}{8} \approx 1.4 \times 10^{12} semantic combinations per phase in Top-128), the remainder of CPT and SFT consists of standard gradient updates where expert FFNs adapt to their active phase bands. A compact 12B12\text{B}-token horizon is the stringent regime because FFNs have fewer post-pruning steps to co-adapt; longer training horizons provide an order of magnitude more gradient updates to absorb the phase boundary and either narrow the remaining quality gap further or sustain tighter KhardK_{\mathrm{hard}} budgets for the same HBM bandwidth and throughput gains.

We appreciate your time and feedback throughout the review process.

Reviewer_NrG1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed Phase-MoE, which combines phase-constrained expert routing learned during training with phase-aware request scheduling during inference to tackle the problem of expert explosion in dLLM. It has identified that across both batch and phases, different experts might be activated; to address that, it proposes phase-conditioned expert pool, soft-to-hard training curriculum, and phase-aware continuous batching. It finetunes Ling-mini and achieves speedup without much performance degradation.

Critical strengths
  1. The system motivations are well-positioned. It is well recognized that batching will diminish the expert sparsity for dLLM denoising phase.
  2. I think the idea of inducing a phase-bounded expert pool is interesting and novel.
  3. Complementary algorithm-system contribution. Training induces phase-local support overlap, and scheduling at inference exploits this characteristic.
  4. The evaluation has clear ablations, such as architecture/scheduler cross comparison and different batch sizes.
Critical weaknesses
  1. The methodology part is filled with heavy math notations and it is a bit hard to read. Specifically, I don’t quite get why 1D Gaussian is used even after reading G.7 in Appendix. Also, it took me a while to understand that the scheduling algorithm in Section 4 depends on the training dynamics from Section 3. I think improving Figure 2 with better illustration will help.
  2. The accuracy loss is not negligible (6~7%). This is understandable as pre-training does not have the same training objective as Phase-MoE and experts are naturally disaggregated with semantics, not phase. I am not sure whether this problem could be solved.
  3. End-to-End serving improvement is not shown. Since the paper’s major motivation is on serving, it is necessary to show the e2e serving performance beyond MoE kernel latency.
Questions for authors

See weakness.

2 comments

We thank Reviewer NrG1 for recognizing our system motivations, phase-bounded expert pools, and algorithm–system co-design. We address your three concerns below, referencing Global Rebuttal §1.1–1.3 and Tables R1–R3.

1. Response to Weakness 1: Intuition for the 1D Gaussian Blur, Co-Design Dependency & Figure 2

We will streamline Section 3.2's notation and update Figure 2 to make the causal link between training (Section 3) and scheduling (Section 4) immediately visual:

  • Why Section 4 Scheduling Requires Section 3 Training: In continuous batching, concurrent requests rarely share identical mask counts (Δϕ=0\Delta\phi = 0). Instead, the Phase-Aware Scheduler (Section 4) groups requests into a narrow window of adjacent phases (e.g., ϕ∈{12,13,14}\phi \in \{12, 13, 14\}, Δϕ=2\Delta\phi = 2). However, batching adjacent phases only saves HBM traffic if adjacent candidate pools (C12\mathcal{C}_{12} and C13\mathcal{C}_{13}) contain nearly the same experts.

  • Physical Role of the 1D Gaussian Blur (Gσ\mathcal{G}_\sigma): Without smoothing across the phases, noisy per-phase token counts in Stage 1 allow adjacent phases ϕ\phi and ϕ+1\phi+1 to lock onto disjoint expert sets. Convolving M>0M_{>0} with a 1D Gaussian kernel along the phase dimension acts as a cross-phase continuity filter: an expert favored at phase ϕ\phi smoothly shares its positive bias with neighbors ϕ±1,ϕ±2\phi \pm 1, \phi \pm 2 (creating the diagonal bands in Figure 3), enforcing expert sharing across adjacent candidate pools Cϕ≈Cϕ±1\mathcal{C}_\phi \approx \mathcal{C}_{\phi \pm 1}.

  • Empirical Proof: Removing the blur (No Gaussian Blur, Table 9) causes adjacent pools to diverge; even under tight PA clustering (Δϕ≈1.04\Delta\phi \approx 1.04), unique expert fetches jump from 112.73 to 148.32 (+31.6%). Similarly, applying PA scheduling to models without smoothed phase pools (Baseline + PA or DES-SEQ + PA, Global Rebuttal Table R2) still activates 171.28–216.67 experts.

  • Figure 2 revision: We will annotate how Section 4's sliding-window scheduler groups requests within a narrow phase range (Δϕ\Delta\phi) and link it to the smoothed diagonal bands in M>0M_{>0} (Section 3), illustrating how overlapping neighbor pools bound batch-level expert activations.

2. Response to Weakness 2: Accuracy Trade-off and Semantic vs. Phase Specialization

Per Global Rebuttal §1.2, we clarify both the accuracy trade-off and semantic preservation:

  • Re-Anchoring on Top-128 (1.3–3.5 pp Gap on 7/8 Tasks, Not 6–7%): Leading with Phase-MoE (Top-128) as our primary recommended operating point cuts unique expert fetches by 44.9% (204.67→112.73204.67 \to 112.73) and A100 MoE latency by 31.8% (20.14→13.74 ms20.14 \to 13.74\text{ ms}) while staying within 1.3 to 3.5 pp of the unconstrained baseline on 7 of 8 standalone benchmarks in Table 1 (−5.0 pp-5.0\text{ pp} on standalone IFEval, and −1.0 pp-1.0\text{ pp} under continuous batching in Table R2).

  • Coexistence of Semantics and Phase (Global §1.2 & Table 9): Phase-MoE forms a 2D hierarchy: (1) Phase 0 Isolation keeps prefill/commits unconstrained across all 256 experts (Table R3); (2) Intra-Phase Semantic Routing dynamically routes each token to k=8k=8 experts via semantic scores sj,is_{j,i} out of Khard=128K_{\mathrm{hard}}=128 candidates per phase; and (3) FFN Co-Adaptation during Soft-to-Hard CPT allows router biases and expert FFNs to co-adapt smoothly before pruning, recovering +24.8 pp on GSM8K (Top-128) and +27.4 pp (Top-96) compared to imposing fixed phase supports from the start of training without Soft Shaping (Direct Hard Mask, Table 9).

3. Response to Weakness 3: End-to-End Serving Performance Beyond MoE Kernel Latency

On an NVIDIA H100 under sustained continuous batching (C=128,Bstep=8,B=32C=128, B_{\mathrm{step}}=8, B=32, Global Rebuttal §1.1 & Table R1), holding the checkpoint fixed and switching FCFS scheduler to PA:

  • Phase-MoE (Top-128): Cuts MoE kernel latency by 22.3% (12.99→10.09 ms12.99 \to 10.09\text{ ms}), driving a +12.8% increase in end-to-end system throughput (161.26→181.85 TPSwall161.26 \to 181.85\text{ TPS}_{\mathrm{wall}}) and 4.4% lower mean request latency (38.54→36.85 s38.54 \to 36.85\text{ s}).

  • Phase-MoE (Top-96): Cuts MoE kernel latency by 24.0% (12.32→9.36 ms12.32 \to 9.36\text{ ms}), yielding +13.3% higher end-to-end throughput (161.99→183.53 TPSwall161.99 \to 183.53\text{ TPS}_{\mathrm{wall}}) and 6.8% lower mean request latency (38.23→35.62 s38.23 \to 35.62\text{ s}) with ≤0.054 ms\le 0.054\text{ ms} CPU scheduling overhead. We will integrate Table R1 into Section 5.4.

We are grateful for your constructive feedback and hope these new results and clarifications address your concerns. Please let us know if any questions remain; if our response resolves the issues raised, we kindly hope you will consider updating your evaluation accordingly.

Thanks for your efforts.

I also read review comments by other reviewers. I agree that there are indeed some concerns in this work.

I will keep my score as I see a few positive things.

Reviewer_aG7FOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Phase-MoE to address the Expert Explosion problem in MoE-based Block Diffusion Language Models (BDLMs) through a co-design of Phase-Constrained MoE and a Phase-Aware Scheduler. During training, Phase-Constrained MoE restricts expert selection based on the denoising phase, defined by the number of remaining masked tokens, using phase-dependent routing biases and a soft-to-hard curriculum to encourage expert sharing across neighboring phases. These constraints are disabled during prefill to preserve model capacity. During inference, the Phase-Aware Scheduler groups requests at similar denoising phases to maximize expert reuse and reduce memory access overhead.

Critical strengths
  1. Unlike existing approaches that simply reduce the number of experts during inference, this paper learns denoising phase-dependent expert specialization and combines it with Phase-Aware Scheduling to address bottlenecks in continuous batching.
  2. The paper evaluates its approach against existing methods across various concurrency levels and batch sizes, and analyzes the individual and combined effects of Phase-Constrained MoE and the Phase-Aware Scheduler.
Critical weaknesses
  1. Lack of End-to-End Evaluation. The paper only measures MoE kernel latency, without demonstrating improvements in actual throughput or end-to-end latency.
  2. Generation Quality Degradation. Top-96 leads to substantial performance drops on GSM8K (−6.4 pp), HumanEval (−7.1 pp), and IFEval (−7.5 pp).
  3. Insufficient Baseline Comparisons. dMoE outperforms the proposed method under certain settings, and the paper does not evaluate existing methods combined with the Phase-Aware Scheduler.
  4. Limited Theoretical Guarantees. The bound (U \leq K_{\mathrm{hard}}) holds only for single-phase batches and is not guaranteed under mixed-phase batching.
Questions for authors

See the Weakness

3 comments

We thank Reviewer aG7F for recognizing the novelty of combining learned phase-dependent expert specialization with Phase-Aware Scheduling to resolve continuous batching bottlenecks. Below, we address each of your four concerns using our new experiments in Global Rebuttal Sections 1.1–1.4 and Tables R1–R3.

1. Response to Weakness 1: End-to-End Throughput and Wall-Clock Latency

We profiled end-to-end wall-clock system throughput (TPSwall\mathrm{TPS}_{\mathrm{wall}}) and per-request latency under sustained continuous batching (C=128C=128, Bstep=8B_{\mathrm{step}}=8, B=32B=32) on an NVIDIA H100 GPU (Global Rebuttal §1.1 & Table R1):

  • Gains from switching schedulers with fixed checkpoint (FCFS→PA\text{FCFS} \to \text{PA}): MoE kernel latency drops by 22.3% on Top-128 (12.99→10.09 ms12.99 \to 10.09\text{ ms}) and 24.0% on Top-96 (12.32→9.36 ms12.32 \to 9.36\text{ ms}), driving a +12.8% end-to-end system throughput gain on Top-128 (161.26→181.85 TPSwall161.26 \to 181.85\text{ TPS}_{\mathrm{wall}}) and +13.3% on Top-96 (161.99→183.53 TPSwall161.99 \to 183.53\text{ TPS}_{\mathrm{wall}}), while reducing mean wall-clock request latency by 4.4% and 6.8%.
  • Gain Over Unconstrained Serving: Relative to Baseline + FCFS, which achieves 169.48 output tokens per second, Top-128 + PA and Top-96 + PA improve end-to-end throughput by +7.3% and +8.3% with CPU scheduling overhead of at most 0.054 milliseconds per step. The Table R1 footnote explains why we compare output tokens per second when checkpoints generate different numbers of tokens. We will add Table R1 to Section 5.4.

2. Response to Weakness 2: Generation Quality and Operating Point Selection (KhardK_{\mathrm{hard}})

Headlining Top-96's hardware numbers in the abstract unintentionally framed Top-96 as our default checkpoint. As detailed in Global Rebuttal §1.2, KhardK_{\mathrm{hard}} is a tunable deployment knob and Phase-MoE (Top-128) is our primary recommended operating point:

  • Top-128 (Recommended Sweet Spot): Cuts unique expert fetches by 44.9% (204.67→112.73204.67 \to 112.73), reduces A100 MoE latency by 31.8% (20.14→13.74 ms20.14 \to 13.74\text{ ms}), and delivers 181.85 output tokens per second (TPSwall\mathrm{TPS}_{\mathrm{wall}}) on H100 while staying within 1.3 to 3.5 pp of the unconstrained baseline on 7 of 8 standalone benchmarks in Table 1 (−5.0 pp-5.0\text{ pp} on IFEval, and −1.0 pp-1.0\text{ pp} under continuous batching in Table R2) with 100% active participation across all 4,864 experts (Table R3). We will re-anchor the abstract and main text on Top-128 and frame Top-96 as a bandwidth-starved boundary.

3. Response to Weakness 3: Evaluating Existing Baselines with Phase-Aware Scheduling

Pairing DES, TEAM, and dMoE with our Phase-Aware (PA) Scheduler at C=64,Bstep∈{8,16}C=64, B_{\mathrm{step}} \in \lbrace 8, 16 \rbrace (Global Rebuttal §1.3 & Table R2) confirms that runtime scheduling alone cannot substitute for learned phase priors:

  • Advantage Over Baseline + PA and DES + PA: Compared with Baseline + PA, Top-96 + PA reduces unique expert fetches by 49.9% with 8 sequences per step and 51.8% with 16. MoE latency falls by 34.0% and 34.4%, respectively. With 16 sequences per step and an average phase spread of about 5.6, Top-96 + PA fetches 48.0% fewer experts than DES-SEQ configured to retain 96 experts (104.52 versus 201.14). Its MoE latency is 31.1% lower (17.79 versus 25.81 milliseconds), with comparable IFEval scores (54.0% versus 53.3%). Top-96 also scores 9.3 percentage points higher on IFEval than DES-BATCH with 96 experts under PA (54.0% versus 44.7%).

  • Scaling Advantage Over dMoE + PA: With 8 sequences per step, dMoE-SEQ + PA fetches fewer experts than Top-96 + PA (85.92 versus 95.39). Increasing to 16 sequences per step raises dMoE-SEQ’s expert fetches by 44.7%, to 124.35. Top-96’s expert fetches increase by 9.6%, to 104.52. At this larger batch size, Top-96 fetches 15.9% fewer experts, has 5.0% lower MoE kernel latency, and achieves a higher IFEval score (54.0% versus 49.7%) than dMoE-SEQ. The dMoE-BATCH variant under PA scores 40.7% on IFEval.

4. Response to Weakness 4: Bounds Under Mixed-Phase Batching (Δϕ>0\Delta\phi > 0)

While U≤KhardU \le K_{\mathrm{hard}} holds deterministically at Δϕ=0\Delta\phi = 0, for a mixed-phase batch spanning [ϕmin⁡,ϕmin⁡+Δϕ][\phi_{\min}, \phi_{\min} + \Delta\phi], telescoping ∣Cϕ+1∖Cϕ∣=Khard1−Jϕ1+Jϕ|\mathcal{C}_{\phi+1} \setminus \mathcal{C}_\phi| = K_{\mathrm{hard}}\frac{1 - \mathcal{J}_\phi}{1 + \mathcal{J}_\phi} (where Jϕ=J(Cϕ+1,Cϕ)\mathcal{J}_\phi = \mathcal{J}(\mathcal{C}_{\phi+1}, \mathcal{C}_\phi) is the pairwise Jaccard similarity) yields the mixed-phase union bound:

U≤∣⋃ϕ=ϕmin⁡ϕmin⁡+ΔϕCϕ∣≤min⁡(N,Khard[1+∑s=1Δϕ1−J(Cϕmin⁡+s,Cϕmin⁡+s−1)1+J(Cϕmin⁡+s,Cϕmin⁡+s−1)])U \le \left\vert \bigcup_{\phi = \phi_{\min}}^{\phi_{\min} + \Delta\phi} \mathcal{C}_\phi \right\vert \le \min\left(N, K_{\mathrm{hard}} \left[ 1 + \sum_{s=1}^{\Delta\phi} \frac{1 - \mathcal{J}(\mathcal{C}_{\phi_{\min}+s}, \mathcal{C}_{\phi_{\min}+s-1})}{1 + \mathcal{J}(\mathcal{C}_{\phi_{\min}+s}, \mathcal{C}_{\phi_{\min}+s-1})} \right]\right)

Although discrete top-k selection does not guarantee a fixed overlap between candidate pools (Jϕ\mathcal{J}_{\phi}), Gaussian smoothing during Stage 1 encourages neighboring phases to share experts between their candidate pools (Cϕ\mathcal{C}_{\phi} and Cϕ+1\mathcal{C}_{\phi+1}), forming the diagonal bands in Figure 3. Empirically, with 128 active requests (C=128C=128) and 8 sequences per step (Bstep=8B_{\mathrm{step}}=8), the average phase spread is approximately 1 (Δϕ=1.03\Delta\phi=1.03). The average number of unique experts fetched (UU) is 89.65 for Top-96 and 112.73 for Top-128, below their respective candidate pool sizes (Khard=96K_{\mathrm{hard}}=96 and 128128; Table 2). With 64 active requests and 16 sequences per step, phase spread increases to 5.59 for Top-96 and 5.66 for Top-128, while average expert fetches remain at 104.52 and 141.70, respectively (Table R2). Removing Gaussian smoothing increases Top-128’s average expert fetches by 31.6%, from 112.73 to 148.32 (No Gaussian Blur, Table 9). We will extend Proposition 1 (Appendix B.3) with a bound that accounts for phase spread and the Jaccard overlap between neighboring candidate pools.

We are grateful for your constructive feedback and hope these new results and clarifications address your concerns. Please let us know if any questions remain; if our response resolves the issues raised, we kindly hope you will consider updating your evaluation accordingly.

Dear Reviewer aG7F,

Thank you again for your thoughtful review and constructive feedback. As the discussion period closes shortly, we wanted to send a brief follow-up to note that we have posted our detailed response above and uploaded the revised manuscript incorporating all requested experiments, analyses, and clarifications.

Please let us know if you have any remaining questions before the window closes---we would be glad to clarify anything further. Thank you again for your time and consideration!

Reviewer_yPFMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies expert memory traffic in block diffusion language models under continuous batching. Although each token activates only a few experts, a batch can collectively access most of the expert pool when its requests are at different denoising stages. Phase-MoE learns a restricted expert pool for each stage during training. At inference time, its scheduler groups requests with similar numbers of remaining masked tokens so they can reuse more of the same experts. The authors evaluate the design on a single NVIDIA A100 and use generation benchmarks and training ablations to measure the quality cost of restricting the expert pool.

Critical strengths

The experiments distinguish the contributions of routing and scheduling. Table 2 provides a four-way comparison with 128 concurrent requests and eight requests per decoding step. Mean MoE kernel latency is 20.14 ms for the unconstrained baseline, 19.07 ms with phase-aware scheduling alone, 17.78 ms with Top-96 phase-constrained routing alone, and 12.38 ms when both are used. This comparison supports the central design choice. Training creates overlapping expert pools across nearby stages, and scheduling allows requests to exploit that overlap within a batch.

The curriculum ablation demonstrates a quality benefit at nearly identical expert usage. In Table 9, imposing a hard Top-96 mask from the start and introducing it through the soft-to-hard curriculum activate an average of 89.85 and 89.65 distinct experts per step, respectively. Yet GSM8K accuracy rises from 25.0% to 52.4% with the curriculum. This is convincing evidence that allowing the router to learn phase specialization before pruning helps preserve model quality.

Critical weaknesses

The tail-latency argument relies on a different baseline from the one used to measure the increase in completion steps. Appendix F.4 compares first-come, first-served (FCFS) and phase-aware scheduling on the same Top-96 checkpoint. Under phase-aware scheduling, mean completion steps fall from 729.73 to 563.87, but P99 completion steps rise from 1,044.83 to 1,445.64. The authors suggest that a 38.5% reduction in kernel latency could offset the additional waiting at the tail. That figure compares the unconstrained model under FCFS with Top-96 under phase-aware scheduling. Holding the checkpoint fixed gives a kernel latency reduction of 30.4%, from 17.78 to 12.38 ms. Even with the matched baseline, mean kernel latency does not establish wall-clock P99 request latency. The kernel speedup is well supported, but the throughput gain and its cost to the slowest requests remain unquantified.

The claim that Phase-MoE breaks the quality–efficiency trade-off is broader than the cross-task results support. At 64 concurrent requests and eight requests per step, DES-BATCH with M=96M=96 and phase-aware Top-96 activate nearly the same number of distinct experts, 94.90 and 95.39, respectively. Phase-MoE has substantially lower additional overhead and higher IFEval accuracy (50.33% versus 46.67%). However, DES scores higher on the other five tasks in Table 5, including GSM8K (64.0% versus 56.0%). The advantage therefore depends on the task and the measure of efficiency. The explanation for dMoE’s performance also needs stronger support. Appendix E.2 describes a separately fine-tuned dMoE baseline, while Appendix E.4 invokes the official checkpoint’s training mixture to infer that its routing overfits to reasoning tasks. The experiments do not isolate this explanation from the effect of imposing batch-wide expert sharing on a checkpoint trained with per-sequence routing.

Questions for authors
  1. For the same Top-96 checkpoint, requests, and generation settings, what are the output throughput in tokens per second and end-to-end P99 request latency in seconds under FCFS and phase-aware scheduling? The configuration in Table 8 with 128 concurrent requests and eight requests per step would be particularly informative.

  2. Which distillation data were used for the dMoE checkpoint evaluated in Table 5? What evidence separates the effect of its training-data distribution from that of switching to batch-wide expert sharing at inference time?

  3. Figure 4 counts the phases in which each expert is eligible for routing. Given the fixed candidate-pool size at each denoising stage and the unrestricted pool at phase zero, the reported mean counts of 17 and 13 follow by construction. Can the authors report actual routed token counts per expert, separately for prefill and denoising, to substantiate the claim that no experts are dead?

2 comments

We thank Reviewer yPFM for the thorough, constructive evaluation and for highlighting our core contributions—specifically, the experimental decoupling of routing and scheduling (Table 2) and the quality preservation achieved via our soft-to-hard curriculum (Table 9). Below, we address remaining questions and concerns, cross-referencing Global Rebuttal Sections 1.1–1.4 and Tables R1–R3.

1. Response to Weakness 1 & Question 1: Matched-Checkpoint End-to-End Serving & P99 Tail Latency

We agree with your critique: comparing the unconstrained baseline under FCFS against Phase-MoE Top-96 under Phase-Aware (PA) scheduling in Appendix F.4 conflated architectural acceleration with scheduling dynamics.

Comparing FCFS and PA on the exact same checkpoint, workload, and decoding settings (C=128,Bstep=8,B=32C=128, B_{\mathrm{step}}=8, B=32) on an NVIDIA H100 (Global §1.1, Table R1):

  • Top-96: Switching FCFS →\to PA cuts MoE latency by 24.0% (12.32→9.36 ms12.32 \to 9.36\text{ ms}; 30.4% on A100, Table 2), boosts throughput by 13.3% (161.99→183.53 TPSwall161.99 \to 183.53\text{ TPS}_{\mathrm{wall}}), and lowers mean request latency by 6.8% (38.23→35.62 s38.23 \to 35.62\text{ s}).
  • Top-128: Switching FCFS →\to PA cuts MoE latency by 22.3% (12.99→10.09 ms12.99 \to 10.09\text{ ms}), boosts throughput by 12.8% (161.26→181.85 TPSwall161.26 \to 181.85\text{ TPS}_{\mathrm{wall}}), and lowers mean latency by 4.4% (38.54→36.85 s38.54 \to 36.85\text{ s}), with ≤0.054 ms/step\le 0.054\text{ ms/step} CPU scheduling overhead.
  • Quantifying Tail Latency (P99): We acknowledge that mean kernel savings do not offset tail queuing delays. Clustering contiguous mask densities causes minority-phase requests to wait until released by τ\tau, increasing P99 request latency by +11.2% (54.97→61.15 s54.97 \to 61.15\text{ s}) for Top-128 and +20.1% (54.67→65.67 s54.67 \to 65.67\text{ s}) for Top-96. We will withdraw the claim in Appendix F.4 that kernel speedups neutralize P99 waiting and explicitly document this throughput-versus-tail trade-off.

2. Response to Question 3: Token Routing Audit (Prefill vs. Denoising)

The average phase counts in Figure 4—17.0 for Top-128 and 13.0 for Top-96—follow directly from the number of candidate experts allowed per phase. These counts describe when experts are available for selection. To check whether they actually receive tokens, we logged routed token counts across all 4,864 layer-expert instances (Global Rebuttal §1.4 and Table R3). This confirms two points:

  • All experts receive tokens in the evaluated workload. All 4,864/4,864 experts receive traffic across Baseline, Top-128, and Top-96. The minimum prefill (Phase 0) and decode (ϕ>0\phi > 0) tokens are, respectively, 93 / 138 for Baseline, 158 / 369 for Top-128, and 99 / 246 for Top-96.
  • Top-128 uses every expert during both Phase 0 and denoising. All 4,864 experts receive tokens in both regimes. During denoising, the fifth percentile of token counts is 5,755, and the median is 37,863. Top-96 uses all 4,654 experts available during denoising (denoising P5 =4,256= 4{,}256; median =40,268= 40{,}268). Its remaining 210 experts (4.32%) are used only during Phase 0, receiving at least 99 tokens each. Among experts available during denoising, the coefficient of variation—the standard deviation divided by the mean token count—decreases from 1.164 for Baseline to 1.013 for Top-128 and 1.069 for Top-96. These values indicate lower relative variation in token counts.

3. Response to Weakness 2 & Question 2: dMoE Disentanglement & Cross-Task Trade-offs

  • Disentangling dMoE Training Data vs. Batch Consensus: We apologize for the ambiguity between Appendices E.2 and E.4. We did not use the official math/reasoning-distilled dMoE checkpoint; we distilled our shared base model on OpenHermes 2.5 using dMoE's block-level top-pp objective (p=0.6p=0.6), and we will correct Appendix E.4. Comparing dMoE-SEQ vs. dMoE-BATCH on that exact same checkpoint in Table 5 (Bstep=16B_{\mathrm{step}}=16) isolates inference-time batch consensus: switching to batch-wide sharing drops GSM8K by −8.0 pp-8.0\text{ pp} (64.0%→56.0%64.0\% \to 56.0\%) and IFEval by −9.33 pp-9.33\text{ pp} (49.33%→40.00%49.33\% \to 40.00\%).
  • Trade-offs vs. DES-BATCH (Tables 5, 6, R2): At Bstep=8B_{\mathrm{step}}=8, DES-BATCH (M=96M=96) beats Top-96 on short MC tasks and GSM8K, but incurs 3.74–3.91 ms/step of voting overhead (vs. 0 ms for Phase-MoE). At Bstep=16B_{\mathrm{step}}=16, our recommended Top-128 beats DES-BATCH (M=96M=96) on GSM8K (66.0% vs. 60.0%, Table 5), while DES-BATCH drops to 44.0% / 44.7% on IFEval (FCFS / PA)—9.3–10.0 pp below Top-96 (54.0%; Global §1.3, Table R2).

We are grateful for your constructive feedback and hope these results and clarifications address your concerns. Please let us know if any questions remain; if our response resolves the issues raised, we kindly hope you will consider updating your evaluation accordingly.

Dear Reviewer yPFM,

Thank you again for your thoughtful review and constructive feedback. As the discussion period closes shortly, we wanted to send a brief follow-up to note that we have posted our detailed response above and uploaded the revised manuscript incorporating all requested experiments, analyses, and clarifications.

Please let us know if you have any remaining questions before the window closes---we would be glad to clarify anything further. Thank you again for your time and consideration!

Comments on the paper

5 comments

1. Global Rebuttal: General Response to All Reviewers

We thank all four reviewers (yPFM, aG7F, NrG1, J2Bm) for their constructive feedback and for recognizing the importance of the continuous batching bottleneck, our phase-aware co-design, and our isolating ablations. Below we summarize our new experiments (Tables R1–R3) and clarifications:

  • §1.1 (Table R1): End-to-end H100 serving telemetry (yPFM, aG7F, NrG1).
  • §1.2: Primary operating point (Top-128) and semantic–phase hierarchy (all reviewers).
  • §1.3 (Table R2): Matched Phase-Aware scheduler comparison against DES, TEAM, and dMoE (aG7F, yPFM, NrG1).
  • §1.4 (Table R3): Token routing audit confirming zero dead experts (yPFM, J2Bm, aG7F).

1.1 Hardware Migration (NVIDIA H100) & End-to-End Serving Telemetry

Hardware Context: The paper evaluations as well as the matched-scheduler baseline comparison (Table R2, matching Table 6) ran on an NVIDIA A100 (80GB). Due to cluster reallocations during the rebuttal period the additional studies (Table R1,3) were run on a single NVIDIA H100 (SXM5 80GB) under identical software settings (dInfer, PyTorch 2.4, BF16). The H100 can load expert weights faster than the A100 because it has higher memory bandwidth (3.35 vs. 2.0 TB/s). Consequently, loading expert weights accounts for less of each step’s total time on the H100 relative to fixed attention and full-vocabulary diffusion sampling (see DiffusionGemma, Taïga et al., 2026). As a result, our H100 end-to-end speedups represent a conservative lower bound compared to workloads with a higher model-to-bandwidth ratio (e.g., larger MoE parameter counts). To isolate the effect of scheduling, we compare schedulers using the same model checkpoint on the same GPU. In the final revision, all hardware tables will report results on the H100.

  • End-to-End Throughput Results: In response to aG7F, NrG1, and yPFM, we measured end-to-end throughput and request latency under multi-domain continuous batching on 900 requests. We kept up to C=128C=128 requests active and processed Bstep=8B_{\mathrm{step}}=8 sequences per step, using denoising blocks of B=32B=32 tokens. Table R1 reports the results.

Table R1: End-to-End Continuous Batching Telemetry on NVIDIA H100 (C=128,Bstep=8C=128, B_{\mathrm{step}}=8)

Model ConfigurationSchedulerUnique Experts (UU)Mean MoE Kernel (ms)System Throughput (TPSwall\mathrm{TPS}_{\mathrm{wall}})Mean Req Latency (s)*P95 / P99 Req Latency (s)Sched. Overhead (ms)
BaselineFCFS204.2313.37169.4830.6337.67 / 45.010.029
BaselinePhase-Aware187.3512.64 (-5.5%)172.42 (+1.7%)28.91 (-5.6%)37.94 / 47.33 (+5.2%)0.052
Phase-MoE (Top-128)FCFS196.0812.99161.2638.5449.28 / 54.970.030
Phase-MoE (Top-128)Phase-Aware115.1210.09 (-22.3%)181.85 (+12.8%)36.85 (-4.4%)52.87 / 61.15 (+11.2%)0.051
Phase-MoE (Top-96)FCFS177.0112.32161.9938.2348.09 / 54.670.032
Phase-MoE (Top-96)Phase-Aware91.529.36 (-24.0%)183.53 (+13.3%)35.62 (-6.8%)54.43 / 65.67 (+20.1%)0.054

Note on cross-checkpoint request latency: Phase-MoE checkpoints generate ∼19.5%\sim 19.5\% more total real output tokens across the workload than the Baseline (45.5k–47.1k45.5\text{k--}47.1\text{k} vs. 39.0k39.0\text{k} tokens) with slightly lower accepted tokens per step (10.4–10.910.4\text{--}10.9 vs. 11.911.9), increasing absolute per-request step counts. Holding the checkpoint fixed and switching first-come, first-served (FCFS) to Phase-Aware scheduler isolates the exact scheduling speedup. We report wall-clock throughput in output tokens per second to account for differences in output volume.

  • Throughput and mean request latency. With the checkpoint held fixed, switching from FCFS to Phase-Aware (PA) scheduling reduced MoE kernel latency by 22.3% to 24.0%. End-to-end throughput, measured in output tokens per second, increased by 12.8% to 13.3%, and mean request latency fell by 4.4% to 6.8%. Compared with the unconstrained Baseline using FCFS, Phase-MoE with PA scheduling achieved 7.3% to 8.3% higher end-to-end throughput over unconstrained Baseline + FCFS.
  • Mean-vs.-Tail Trade-off: Because the Phase-Aware scheduler groups requests with similar mask densities, isolated minority-phase requests wait until released by the dynamic starvation trigger τ\tau, increasing P99 request latency by +11.2% for Top-128 and +20.1% for Top-96. We will revise Appendix F.4 to explicitly report this mean-versus-tail trade-off rather than suggesting kernel speedups fully neutralize P99 waiting.

1.2 Re-Anchoring on Top-128 & Preserving Semantic Specialization

Reviewers aG7F, NrG1, J2Bm, and yPFM raised questions regarding Top-96's accuracy degradation and how phase-conditioned routing preserves semantic capacity. We clarify two core principles.

Top-128 is our recommended configuration. Highlighting Top-96’s hardware results in the abstract unintentionally made it appear to be our default model. Our main configuration is Phase-MoE with Top-128. With up to 128 active requests and 8 sequences processed per step, it reduces the average number of unique experts fetched by 44.9%, from 204.67 to 112.73. It also reduces MoE latency on the A100 by 31.8%, from 20.14 to 13.74 milliseconds, and achieves an end-to-end throughput of 181.85 output tokens per second on the H100.

Top-128 scores 1.3 to 3.5 percentage points below the unconstrained baseline on seven of the eight standalone benchmarks in Table 1. The gaps are 1.6 percentage points (pp) on ARC-C, 1.3 on ARC-E and PIQA, 1.7 on HellaSwag, 3.5 on MMLU and GSM8K, and 3.2 on HumanEval. On IFEval, the gap is 5.0 pp when evaluated alone and narrows to 1.0 pp under continuous batching (Table R2).

Top-96 makes a more aggressive trade-off to reduce memory traffic. In our simplified training setup, which uses 12 billion tokens and omits reinforcement learning and confidence-sharpening losses, its more limited expert capacity affects performance on long generated sequences (Appendix C.1). We will revise the abstract and main text to present Top-128 as our recommended configuration.

  • How Mask Density (ϕ\phi) Complements Rather Than Replaces Token Semantics (Addressing Reviewers J2Bm and NrG1 on why conditioning routing on phase does not compromise token semantics):
  1. Information Regime Shift: Early in denoising, most positions in the current block are masked, so the model relies mainly on the prompt and previously generated blocks. As denoising progresses, more tokens within the block become available as context. Prior work has observed that early steps tend to shape the overall meaning, while later steps refine the wording (Wu et al., 2026). In Stage 1, we initialize all expert–phase biases to zero (Mϕ,j=0M_{\phi,j}=0) and learn them from training data. Figure 6 shows that normalized mutual information, which measures the association between expert selection and denoising phase, rises from approximately zero to 0.08.
  2. 2D Semantic–Phase Hierarchy: First, Phase 0 Isolation (ϕ=0\phi=0) leaves prompt prefill and block commits 100% unconstrained across all N=256N=256 experts (Table R3). Restricting expert access during this phase reduces GSM8K accuracy by 16.1 percentage points (Table 9). Second, during denoising (ϕ>0\phi \gt 0), Top-128 retains a pool of Khard=128K_{\mathrm{hard}}=128 candidate experts per phase. Each token is then dynamically routed to k=8k=8 experts within that pool via its gating score$s_{j,i}$. This leaves (1288)≈1.4×1012\binom{128}{8} \approx 1.4 \times 10^{12} possible semantic combinations of experts.
  3. FFN co-adaptation during Soft Shaping (Table 9). The Direct Hard Mask ablation (Appendix G.5) imposes a fixed expert pool for each phase using activation frequencies from the initial autoregressive model. It applies these restrictions from the start of continual pre-training and skips Stage 1 Soft Shaping. This prevents access to some pretrained experts before the model can adapt, reducing GSM8K accuracy to 30.5% for Top-128 and 25.0% for Top-96. Gradually learning the phase biases, M>0M_{\gt 0}, before hard pruning allows the router biases and expert feed-forward networks (FFNs) to co-adapt smoothly. GSM8K accuracy consequently improves from 30.5% to 55.3% (+24.8 pp) for Top-128, and from 25.0% to 52.4% (+27.4 pp) for Top-96.

1.3 Comparing test-time baselines under Phase-Aware scheduling

Reviewers aG7F, yPFM, and NrG1 asked whether existing test-time methods (DES, TEAM, and dMoE) achieve similar gains with our Phase-Aware (PA) scheduler. We evaluated all baselines with PA scheduling, keeping up to 64 requests active (C=64C=64) and processing either 8 or 16 sequences per step (Bstep∈{8,16}B_{\mathrm{step}} \in \{8, 16\}).

Table R2: Baselines Combined with Phase-Aware Scheduling (C=64C=64, A100)

Model & InterventionBstepB_{\mathrm{step}}Phase Spread (Δϕ\Delta\phi)Unique Experts (UU)MoE Latency (ms)IFEval Score (%)
Baseline + PA8 / 162.26 / 5.77190.53 / 216.6719.18 / 27.1251.7% / 52.0%
DES-SEQ (M=128M=128) + PA8 / 162.25 / 5.69176.45 / 204.8218.20 / 26.1549.7% / 50.3%
DES-SEQ (M=96M=96) + PA8 / 162.24 / 5.68171.28 / 201.1417.84 / 25.8150.0% / 53.3%
DES-SEQ (M=64M=64) + PA8 / 162.23 / 5.66156.12 / 188.7016.58 / 24.6249.3% / 52.0%
DES-BATCH (M=128M=128) + PA8 / 162.25 / 5.67124.30 / 126.5514.40 / 18.8548.7% / 47.3%
DES-BATCH (M=96M=96) + PA8 / 162.24 / 5.6794.18 / 95.1212.50 / 15.9646.3% / 44.7%
DES-BATCH (M=64M=64) + PA8 / 162.23 / 5.6563.45 / 63.7010.84 / 12.7843.3% / 42.7%
TEAM / LAC-SEQ + PA8 / 162.21 / 5.64107.64 / 144.2019.52 / 27.6548.3% / 47.3%
TEAM / LAC-BATCH + PA8 / 162.22 / 5.64106.15 / 142.9019.55 / 27.5848.7% / 49.0%
dMoE-SEQ (P=0.6P=0.6) + PA8 / 162.25 / 5.6985.92 / 124.3511.95 / 18.7249.0% / 49.7%
dMoE-BATCH (P=0.6P=0.6) + PA8 / 162.24 / 5.6838.60 / 45.159.98 / 11.1243.3% / 40.7%
Phase-MoE (Top-128) + PA8 / 162.17 / 5.66118.37 / 141.7014.18 / 20.3550.0% / 51.0%
Phase-MoE (Top-96) + PA8 / 162.22 / 5.5995.39 / 104.5212.66 / 17.7950.3% / 54.0%
  • Selecting experts separately for each sequence still leads to high memory traffic (SEQ + PA). Even under identical phase alignment (Δϕ≈2.2\Delta\phi \approx 2.2 at Bstep=8B_{\mathrm{step}}=8, ≈5.6\approx 5.6 at Bstep=16B_{\mathrm{step}}=16), test-time SEQ methods lack trained phase support pools, so semantically diverse prompts select different expert sets. With 16 sequences per step (Bstep=16B_{\mathrm{step}}=16), DES-SEQ configured to retain 96 experts fetches an average of 201.14 unique experts, while dMoE-SEQ fetches 124.35. Doubling the number of sequences per step increases expert fetches by 44.7% for dMoE-SEQ, compared with 9.6% for Top-96.
  • Batch-wide consensus still degrades quality (BATCH + PA): Forcing a single shared coreset across semantically diverse requests caps unique experts (UU), but still degrades IFEval accuracy even when denoising phases are aligned—dropping to 44.7% for DES-BATCH (M=96M=96) + PA and 40.7% for dMoE-BATCH + PA at Bstep=16B_{\mathrm{step}}=16, compared to 54.0% for Phase-MoE (Top-96) + PA (consistent with the multi-task degradation in Table 5).
  • Co-Design outperforms baselines+PA: Against Baseline + PA, Top-96 + PA cuts unique experts by 49.9% / 51.8% and latency by 34.0% / 34.4% (Bstep=8/16B_{\mathrm{step}}=8/16). At Bstep=16B_{\mathrm{step}}=16, Top-96 + PA uses 48.0% fewer experts and achieves 31.1% lower MoE latency than DES-SEQ (M=96M=96) + PA at comparable IFEval (54.0% vs. 53.3%), and outperforms dMoE-SEQ + PA across expert fetches (104.52104.52 vs. 124.35124.35), kernel latency (17.7917.79 vs. 18.72 ms18.72\text{ ms}), and IFEval (54.0% vs. 49.7%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.