acceptodds
Under review as a conference paper at ICLR 2027

Post-Trained MoE Can Skip Half Experts via Self-Distillation

Abstract

Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary experts during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 11.7 and 14.9 points on the two models, and delivers ~1.20× end-to-end inference speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Dynamic MoE, Self-Distillation, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6xdbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ZEDA, a framework for converting post-trained MoE models into dynamic ones. It combines zero-output experts, group-level balancing, and two-stage self-distillation from the original model to reduce expert computation while preserving performance. Experiments on two MoE models show over 50% less expert computation and roughly 20% higher inference throughput, with a small drop in average benchmark scores.

Critical strengths
  1. The paper is clearly written and easy to follow.

  2. I appreciate the honest reporting of practical speedups. In my own experiments, expert pruning has also delivered only modest gains with the highly optimized fused MoE kernels in vLLM and SGLang. The reported results are more credible to me than some of the large speedup claims in related work.

Critical weaknesses
  1. I agree with using OPD to recover performance after reducing expert computation, but I have reservations about the SFT stage. Learning from a fixed set of teacher-generated responses may encourage memorization and hurt generalization. I find OPD a cleaner approach and would like to see further evaluation of generalization to unseen domains.

  2. Although Qwen3-30B-A3B remains widely used, I would encourage the authors to include experiments on more recent MoE models.

  3. For GLM-4.7-Flash, I would also like to see evaluations on agentic benchmarks such as SWE-bench, to assess whether the adaptation preserves the model’s agent capabilities.

  4. sft / opd is both widely used post-training tech in practice, this paper used them in expert-pruning, which may reduce novelty.

Questions for authors
  1. How does ZEDA compare with SlimQwen? I understand that the two are concurrent works, so I have not treated the lack of a comparison as a weakness, but I would appreciate the authors’ views on their main similarities and differences.

  2. In my own experiments on dynamic expert pruning, renormalizing the remaining experts’ routing weights worked better, whereas Appendix B.6 reports the opposite result. How do the authors explain this discrepancy? In particular, could the preferred choice depend on whether the model undergoes further training after pruning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6xdb (1/2)

We thank the reviewer for the valuable comments and constructive suggestions.

W1: The Role of SFT and Generalization to Unseen Domains

Appendix B.3 provides evidence for the importance of the SFT stage. Table 6 (p. 18) compares SFT-only, OPD-only, and SFT→OPD under comparable total adaptation compute (the single-stage variants receive additional training steps so that their cost matches or exceeds that of the full pipeline). OPD-only achieves the lowest average accuracy, 72.9, compared with 73.6 for SFT-only and 74.2 for SFT→OPD. Our interpretation is that SFT first establishes stable zero-expert routing, whereas OPD without this initialization must simultaneously learn routing decisions and generate coherent on-policy rollouts. This makes the initial adaptation more challenging.

We agree that evaluation on unseen domains is important for assessing whether adaptation preserves capabilities beyond the training domains. We therefore additionally evaluate Qwen3-30B-A3B, both NET variants, ZEDASFT\mathrm{ZEDA}_{\mathrm{SFT}}, and ZEDA on four unseen domain tasks (BFCL V3, MMLU, GPQA, and ARC-c). The results are shown in the table below. ZEDA achieves only 0.5 percentage points below the original model, while skipping approximately 50% of routed-expert computation. It outperforms all baselines. These results support capability retention in the evaluated unseen domains.

MethodAvg AccAvg rZEr_{\mathrm{ZE}}BFCL V3MMLUGPQAARC-c
Qwen3-30B-A3B80.10.070.490.163.396.5
NETSFT\mathrm{NET}_{\mathrm{SFT}}78.450.067.589.160.996.2
NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}78.650.068.388.960.896.4
ZEDASFT\mathrm{ZEDA}_{\mathrm{SFT}}78.849.867.789.262.296.3
ZEDA79.650.069.589.263.296.4

We would also clarify the intended use of ZEDA. When a deployment domain is specified, representative prompts from that domain can be included in the self-distillation corpus. Our objective is to retain capabilities while reducing computation, and it does not require substantial positive transfer to every domain excluded from adaptation. The additional results above nevertheless provide evidence that retention extends beyond the domains explicitly targeted during adaptation.

W2: Validation on a More Recent MoE Backbone

Following the reviewer's suggestion, we extend the experiments to the more recent Qwen3.5-35B-A3B, following the adaptation and evaluation settings used for Qwen3-30B-A3B. The results are shown in the table below.

MethodAvg AccAvg rZEr_{\mathrm{ZE}}AIME 24AIME 25AIME 26GSM8KMATH-500LCB v5LCB v6HumanEval+MBPP+IFBenchIFEval
Qwen3.5-35B-A3B82.80.089.683.587.593.394.873.068.391.874.965.389.1
AdaMoE71.052.372.062.076.588.686.053.547.289.771.154.080.4
Dynamic Skipping73.549.881.669.077.388.487.655.152.886.773.357.379.3
NETSFT\mathrm{NET}_{\mathrm{SFT}}78.050.086.776.585.489.190.464.959.790.775.059.380.8
NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}79.750.086.977.084.489.694.468.162.590.873.366.083.9
ZEDASFT\mathrm{ZEDA}_{\mathrm{SFT}}80.053.788.480.685.688.392.067.563.991.573.761.786.1
ZEDA82.050.988.580.787.192.694.471.166.691.975.865.388.5

ZEDA achieves 82.0 average accuracy, compared with 82.8 for the original model, while skipping 50.9% of routed-expert computation. It achieves the highest average score among all baselines. These results extend the evidence for ZEDA's effectiveness to a more recent MoE backbone.

AuthorsOpenReview ↗

Response to Reviewer 6xdb (2/2)

W3: Preservation of Agentic Capabilities

We agree that agentic benchmarks provide an important additional test of capability retention. We evaluate the original GLM-4.7-Flash, NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}, and ZEDA on SWE-bench Verified, following the GLM team's reported evaluation settings and the default OpenHands configuration. The results are shown in the table below. Compared to the original model, the ZEDA model scores slightly lower on SWE-bench Verified, significantly outperforming NET, while skipping 55.5% of routed-expert computation.

MethodSWE-bench VerifiedAvg rZEr_{\mathrm{ZE}}
GLM-4.7-Flash57.40.0
NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}52.050.0
ZEDA56.155.5

We further evaluate three models on another, slightly simpler agentic benchmark, BFCL V3. The results are shown in the table below. On these benchmarks, ZEDA continues to outperform NET, while showing almost no performance gap compared to the original model. Together, these evaluations support the preservation of the model's agent capabilities.

MethodBFCL V3Avg rZEr_{\mathrm{ZE}}
GLM-4.7-Flash73.70.0
NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}71.950.0
ZEDA73.549.8

W4: Novelty

We agree that SFT and OPD are established techniques, and our novelty claim does not rest on them. We study the practically important problem of cost effectively converting a mature post trained top-K MoE into a dynamic expert model without repeating pretraining or the full post training pipeline, and to our knowledge, we are the first to successfully achieve this.

The main technical contributions are the Group Auxiliary Loss, which controls relative utilization between normal and zero experts without explicitly imposing uniform routing among normal experts, the validation of zero-expert injection as an effective conversion mechanism, an empirical characterization of the resulting computation dynamics, and a detailed analysis of alternative zero-computation experts that motivates the choice of zero-output experts.

Q1: Similarities and Differences Between ZEDA and SlimQwen

Both works seek to reduce computation while largely preserving performance, but target different stages of the model lifecycle.

ZEDA primarily targets MoE models that have undergone the complete pre-training and post-training pipeline (referred to as post-trained MoE models in our paper). These models have typically undergone SFT, RL, and OPD and are ready for deployment in inference applications. In contrast, SlimQwen focuses on compressing base models that have only undergone pre-training. As explicitly indicated by its title, “in Large MoE Model Pre-training,” both the methodological design and experimental evaluations of SlimQwen primarily focus on base models that have undergone pre-training but not subsequent post-training stages.

In terms of specific methodological design and implementation details, the two approaches share little similarity. We acknowledge that our paper lacks a discussion and citation of SlimQwen, and we will address this omission in the revised version of the paper.

Q2: Renormalization

We conduct an additional training-free experiment on Qwen3-30B-A3B. We apply only zero-expert injection, and evaluate the resulting model with and without renormalizing the remaining normal experts' routing weights. No SFT or OPD is performed. The results are shown in the table below.

MethodAvg AccAvg rZEr_{\mathrm{ZE}}AIME 24AIME 25AIME 26GSM8KMATH-500LCB v5LCB v6HumanEval+MBPP+IFBenchIFEval
w/ Renorm64.544.956.044.056.095.592.841.839.680.177.241.384.8
w/o Renorm64.345.756.341.755.296.092.643.538.980.176.842.583.2

In this training-free setting, the average accuracies are close, with only a 0.2-point difference, and the non-renormalized variant shows no clear advantage. These results are consistent with the reviewer's suggestion that the preferred choice may depend on whether further training is performed. We therefore view the two observations as compatible results from different adaptation regimes, and limit our recommendation to omit renormalization to the tested adaptation setting.

Thank you for the effort you have put into the rebuttal. However, my concerns about using SFT to recover the capabilities of pruned models remain. I do not question the effectiveness or importance of SFT. Rather, I believe that such recovery training would be better integrated into pre-training as a form of “expert-pruning-aware training,” analogous to quantization-aware training (QAT). In the current pipeline, post-pruning SFT cannot cover the full diversity of the original training data, leaving concerns about overfitting to the SFT data and incomplete recovery of the model’s capabilities. I do not think the current response adequately addresses this issue.

I also continue to find the paper’s overall novelty limited. That said, I appreciate the insightful discussion of renormalization, from which I learned a lot.

Taking everything into consideration, I have decided to maintain my original score.

Thank you for considering our response. We would like to further clarify the questions you raised.

ZEDA is framed around, and specifically aims at, converting MoE models that have completed the full post-training pipeline into more efficient dynamic MoEs. Pretraining itself is outside ZEDA's intended setting. ZEDA retains all original model parameters. Since ZEDA does not perform parameter pruning, intuitions and empirical observations about overfitting in that setting may not apply directly to our approach.

Our additional experiments further show no clear signs of overfitting to the SFT data or substantial shortfalls in capability recovery on the evaluated benchmarks. On BFCL V3, MMLU, GPQA, and ARC-c, Qwen achieves 79.6 versus the original model's 80.1 average accuracy, while skipping 50% of routed-expert computation. These results provide evidence against recovery merely reflecting SFT memorization.

We respectfully view the conversion framework, Group Auxiliary Loss, and analyses of expert types, renormalization, and dynamic allocation as meaningful contributions. ZEDA's gains over NET under the same SFT→OPD stages further support the value of its design beyond established training techniques.

If any questions remain or further clarification would be helpful, please do not hesitate to let us know. We would be happy to continue the discussion. Thank you for your time and engagement.

Reviewer_xZMWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to reduce expert computation in an already post-trained MoE without repeating its original training pipeline. ZEDA adds zero-output experts to the routing pool while keeping the number of selected experts unchanged, so selecting a zero expert replaces a normal expert computation. The modified model is adapted using teacher-generated supervised fine-tuning followed by on-policy self-distillation from the original model. A group auxiliary loss controls the balance between normal and zero experts without explicitly imposing uniform utilization within the normal-expert group. Experiments cover Qwen3-30B-A3B and GLM-4.7-Flash on 11 math, coding, and instruction-following benchmarks. The reported average scores decrease by 0.7 percentage points on each model, while the average zero-expert activation ratios reach 51.2% and 53.0%. SGLang measurements show roughly 16-26% throughput improvements across prefill, decode, and the tested single-GPU and expert-parallel settings.

Critical strengths
  1. The paper addresses a useful deployment problem and reports a meaningful practical result. Halving active expert computation in a model that has already completed post-training is more relevant to many users than training a new dynamic MoE from scratch. The result is demonstrated on two model families, and the measured throughput gains show that the reduction is not merely a FLOP-count argument. The adaptation time is also reported with teacher-data generation included, which makes the cost easier to assess.

  2. The ablations are substantially more informative than a comparison against the original model alone. NET with SFT followed by OPD is a relevant fixed-budget control. Appendix B.3 compares the two-stage pipeline with single-stage alternatives at comparable or greater compute, and Appendix B.4 checks continued distillation of the unmodified model. Table 5 is especially useful: at similar zero-expert utilization, group balancing raises the math average from 59.5 to 82.2 relative to expert-level balancing. Appendix B.5 also reports repeated results rather than relying entirely on one evaluation. Together, these experiments make the main adaptation result credible.

Critical weaknesses
  1. The headline comparison does not fully convey the gain over the most relevant control. The 11.7- and 14.9-point improvements are over the named dynamic-routing baselines. The much closer comparison is NET with SFT followed by OPD, which obtains 73.0 versus ZEDA's 74.2 on Qwen and 72.8 versus 73.9 on GLM in Table 1. This remains a useful improvement, particularly because ZEDA skips slightly more experts, but it is a different contribution in magnitude. Section 3.1 states that AdaMoE uses the same training hyperparameters and Dynamic Skipping follows its original adaptation procedure; it does not make teacher supervision, SFT/OPD exposure, adaptation compute, and tuning effort fully comparable across these methods. Moreover, Figure 2 measures speed against the original model rather than NET. The paper therefore establishes the value of the overall conversion pipeline more clearly than the advantage of its dynamic routing over a well-adapted fixed-budget alternative at actual serving cost.

  2. The serving measurements cover a narrow operating point for the end-to-end efficiency claim. Appendix C specifies 256 examples sampled from the adaptation corpus, maximum concurrency 32, and sequence length 8192, with input and output token counts controlled across models. This is a useful controlled benchmark, and the single-GPU and EP=4 measurements are strengths. However, these results do not show how the gain changes with short interactive requests, long reasoning outputs, different concurrency, or naturally varying response lengths. The statement that the models produce similar response lengths is not accompanied by an original-versus-ZEDA length comparison; Table 3 reports only ZEDA's lengths. Prefill and decode throughput also do not by themselves establish request-level latency improvements under realistic workloads. The current evidence supports a roughly 20% gain in the measured settings, with a less certain benefit outside them.

  3. The repeated-run evidence is useful but does not clearly establish training stability across both models. Appendix B.5 should be credited: Table 8 reports three Qwen runs, including average accuracy of 74.0 +/- 0.17 for ZEDA and 72.9 +/- 0.14 for NET with SFT followed by OPD. However, the text calls these "three evaluation runs," leaving it unclear whether the entire adaptation was retrained or a fixed checkpoint was sampled repeatedly. These measure different sources of variation, especially because zero-expert router initialization and on-policy rollouts are stochastic. The table also does not cover GLM, the two external dynamic baselines, or the original model. The avg@32 and avg@8 protocols reduce sampling noise but do not replace independent training seeds. Thus, the available statistics support repeatability within the reported Qwen protocol more directly than the broader claims of adaptation robustness and capability retention.

  4. A small macro-average loss masks some meaningful task-level regressions. Table 1 reports only a 0.7-point average drop on each model, but Qwen's LiveCodeBench v5 and v6 scores fall by 3.3 and 3.9 points. GLM drops by 4.4 points on AIME 24, 3.4 on AIME 25, and 2.7 on HumanEval+. Improvements elsewhere offset these losses in the mean. This may be an acceptable trade-off for substantial compute savings, but it is weaker than preserving established capabilities uniformly. The evaluation is also concentrated on the same broad categories represented in the adaptation corpus. I would distinguish preservation on these benchmarks from general preservation of post-training capabilities. Per-task uncertainty and an explicit accuracy-loss tolerance would make the practical conclusion much easier to assess.

  5. The proposed explanation of routing behavior is not yet established by the evidence. Group balancing avoids explicitly enforcing uniform use of individual normal experts, but that does not ensure that their original relative routing is preserved during adaptation. Table 5 demonstrates a performance advantage over the tested expert-level loss, not the preservation of the original routing distribution itself. Similarly, Figure 3 pools tokens from 110 responses and relates skipping to entropy and teacher-student log-probability differences. These are correlations measured after adaptation; response position, token type, and domain can influence all of them. Table 3 also finds similar skipping ratios across human-annotated difficulty levels, which appropriately limits the interpretation to token-level effects. The NET control supports the overall value of a dynamic design, but it does not show that the particular proposed difficulty signals explain that value. Direct routing-drift measurements and a targeted compute-matched intervention would strengthen the mechanistic claims.

  6. The role of held-out validation in selecting the adaptation recipe is unclear. Section 4.2 uses average benchmark scores to motivate the 60k-prompt setting, and Section 4.3 uses benchmark performance and utilization to select w=2. Appendix B.7 also examines several auxiliary-loss coefficients. The paper does not state whether these analyses were conducted on a separate validation split, whether the settings were fixed before final evaluation, or how checkpoints were selected. Appendix C's check for train-evaluation sample overlap is valuable, but it addresses a different issue from using final benchmark scores to choose the recipe. Since several relevant margins are around one point, clarity about model selection matters. This is a protocol question, not evidence that leakage occurred.

Questions for authors

The questions below correspond to weaknesses 1-6 and concern evidence that could change my assessment of the current claims.

  1. Please specify teacher supervision, adaptation stages, training tokens or GPU-hours, and tuning budgets for AdaMoE and Dynamic Skipping. How much of the gap remains when the closest compatible baseline receives comparable self-distillation? Could you also compare ZEDA and NET with SFT followed by OPD using measured latency or throughput, so that their quality-efficiency trade-off is clear?

  2. Does the speedup persist at shorter and longer input/output lengths and at low as well as high concurrency? Please provide at least a small sensitivity study and an original-versus-ZEDA response-length comparison on held-out prompts. Can you report request-level latency or time to first token alongside throughput, with repeated timing measurements and uncertainty?

  3. Do the three runs in Table 8 independently repeat initialization, data sampling, SFT, and OPD, or only evaluation? Please identify the seeds and what was varied. If they are evaluation-only runs, please add at least three independent adaptation seeds for the key ZEDA-versus-NET comparison on both backbones and report mean +/- standard deviation for accuracy and expert utilization. The original model should also have repeated evaluations to quantify the uncertainty in the retention comparison.

  4. Are the larger LiveCodeBench and AIME drops reproducible, and what task-level losses would the authors consider acceptable for the claimed capability preservation? Please provide uncertainty estimates for the principal regressions and clarify the scope of the preservation claim. Do any existing held-out evaluations outside the adaptation categories support its broader wording?

  5. How much does routing among normal experts change from the original model under group balancing versus expert-level balancing? Can you quantify that change and test whether the token-level associations remain after controlling for response position and token type? A compute-matched control that disrupts the learned tokenwise allocation, while retaining the normal-expert selection rule, would help distinguish useful allocation from an incidental correlation; is such evidence available?

  6. Which data or validation split were used to choose prompt count, w, alpha, adaptation duration, and checkpoints? Were the final 11 benchmark results consulted during those decisions? Please separate validation-based selection from final testing and state whether the same selection protocol was applied to the baselines.

9 comments
AuthorsOpenReview ↗

Response to Reviewer xZMW (1/6)

We thank the reviewer for the valuable comments and detailed, constructive questions.

W1 & Q1: Baseline Comparability and the Quality–Efficiency Trade-off

We acknowledge that the presentation of the improvements should be more precise. These numbers compare ZEDA with existing dynamic-routing methods and do not include NET, which we introduced as a variant of ZEDA. We will revise the manuscript to distinguish these comparisons explicitly.

Regarding adaptation cost, AdaMoE builds on SFT and uses exactly the same SFT dataset as ZEDA. Its training settings, including data volume, batch size, training steps, and learning rate, match those of ZEDA's SFT stage. Thus, teacher supervision, adaptation stages, training tokens or GPU-hours, and tuning resources are approximately matched between AdaMoE and ZEDA_{SFT}. The advantage of ZEDA_{SFT} over AdaMoE therefore supports the effectiveness of ZEDA's design. Dynamic Skipping is a training-free method that uses the same SFT dataset for calibration. Its training GPU-hour cost can be regarded as essentially zero relative to ZEDA. AdaMoE and Dynamic Skipping are the very few relevant baselines that we were able to identify after an extensive search. Applying ZEDA's SFT→OPD objective to them would instead define a new hybrid method.

The closest compatible control is NET, which undergoes the full SFT→OPD pipeline. Its teacher supervision, adaptation stages, training tokens or GPU-hours, and tuning resources are approximately matched to ZEDA. ZEDA's consistent gains over NET support the effectiveness of dynamic expert allocation. Furthermore, Appendix B.4 reports the results of applying exactly the same SFT and OPD procedure to the original model, showing that the same self-distillation procedure alone provides no extra accuracy gain for the unmodified model.

We add a direct comparison with NET (SFT→OPD), the fixed top-4 control. All timing measurements use one H200, SGLang, BF16, 256 requests, and 3 repetitions of the same workload. Inputs and per-request output lengths are identical across methods. The reference workloads use concurrency 32 and a nominal mean length of 8k tokens for prefill inputs and decode outputs. We also apply request warmup and clear the KV cache for all methods before timing. We measure request-level end-to-end latency (E2E) and time to first token (TTFT) in addition to throughput. The table below reports mean ± sample SD over three runs.

MethodAvg Acc ↑Expert skipping (%)Prefill (tokens/s) ↑Decode (tokens/s) ↑TTFT (s) ↓E2E (s) ↓
Qwen3-30B-A3B74.90.045,271 ± 961,752.4 ± 1.4458.42 ± 0.46601.81 ± 0.54
NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}73.050.055,003 ± 282,028.6 ± 1.7397.13 ± 0.24521.10 ± 0.35
ZEDA74.251.254,262 ± 1562,113.0 ± 2.5382.11 ± 0.37500.94 ± 0.50

Compared with NET, ZEDA achieves higher accuracy, higher decode throughput, and lower TTFT and E2E latency. These measurements directly demonstrate ZEDA's favorable quality–efficiency trade-off in the tested serving setting.

AuthorsOpenReview ↗

Response to Reviewer xZMW (2/6)

W2 & Q2: Speedup in Different Settings and Response Length

To evaluate the speedup more comprehensively, we vary input/output lengths and concurrency, reporting request-level E2E latency and TTFT alongside throughput. Each setting is measured three times using the timing protocol described above. The following tables report paired speedups as mean ± sample SD over three runs, ZEDA/Original for throughput and Original/ZEDA for latency, so values above 1 indicate improvement. The first table varies the nominal mean length at concurrency 32, and the second varies concurrency at a nominal mean length of 8k tokens.

Mean length (tokens)Prefill throughput speedup ↑Decode throughput speedup ↑TTFT speedup ↑E2E speedup ↑
4k1.2526 ± 0.03381.2456 ± 0.00321.2379 ± 0.00681.2395 ± 0.0054
6k1.2457 ± 0.04011.2317 ± 0.00311.2269 ± 0.00351.2277 ± 0.0034
8k1.1986 ± 0.00521.2058 ± 0.00241.1997 ± 0.00221.2014 ± 0.0022
10k1.1877 ± 0.00641.2087 ± 0.00121.2024 ± 0.00181.2036 ± 0.0016
ConcurrencyPrefill throughput speedup ↑Decode throughput speedup ↑TTFT speedup ↑E2E speedup ↑
161.2287 ± 0.05431.2501 ± 0.00301.2388 ± 0.00151.2405 ± 0.0011
321.1986 ± 0.00521.2058 ± 0.00241.1997 ± 0.00221.2014 ± 0.0022
481.2002 ± 0.00131.1767 ± 0.00021.1676 ± 0.00051.1697 ± 0.0003

ZEDA consistently accelerates inference across all tested lengths and concurrency levels, with gains that are large relative to the observed timing variability. The roughly 1.2× speedup extends beyond throughput to request-level E2E latency and TTFT. These results broaden the evidence for ZEDA's inference efficiency across the evaluated workloads.

We additionally compare the original model's and ZEDA's response lengths. We measure mean response length on the 11 in-domain tasks and on held-out prompts from four additional out-of-domain tasks: BFCL V3, MMLU, GPQA, and ARC-c. The results are shown in the table below.

ModelMean response length: in-domain tasks (tokens)Mean response length: held-out prompts (tokens)
Qwen3-30B-A3B8,485.42,433.2
ZEDA8,506.62,154.8

The in-domain mean response lengths are nearly identical, differing by approximately 0.25%. On held-out prompts, ZEDA's responses are approximately 11.4% shorter. We therefore do not observe substantially longer responses from ZEDA in either setting.

AuthorsOpenReview ↗

Response to Reviewer xZMW (3/6)

W3 & Q3: Independent Adaptation Runs and Evaluation Variability

The three runs in Table 8 are repeated evaluations using different evaluation seeds, 42, 21, and 63. We will make this explicit in the revised manuscript. To additionally assess randomness throughout adaptation, we conduct three independent full-pipeline runs for ZEDA, ZEDASFT\mathrm{ZEDA}_{\mathrm{SFT}}, NETSFT\mathrm{NET}_{\mathrm{SFT}}, and NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}} on both backbones. Each run independently repeats all applicable stages, including initialization, data sampling, SFT, OPD, and evaluation, with the run seed consistently set to 42, 21, or 63. The table below reports mean ± SD in percent.

MethodAvg AccAvg rZEr_{\mathrm{ZE}}AIME 24AIME 25AIME 26GSM8KMATH-500LCB v5LCB v6HumanEval+MBPP+IFBenchIFEval
Qwen3-30B-A3B
NETSFT\mathrm{NET}_{\mathrm{SFT}}72.3 ± 0.1850.0 ± 0.0076.8 ± 0.8465.7 ± 1.4671.3 ± 1.4694.7 ± 0.3094.3 ± 0.9255.9 ± 1.9451.5 ± 1.2487.0 ± 1.6478.7 ± 0.8436.7 ± 2.0082.7 ± 0.78
NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}72.9 ± 0.3850.0 ± 0.0078.1 ± 2.4867.2 ± 1.6271.3 ± 1.4695.3 ± 0.3694.4 ± 0.7056.5 ± 1.2252.4 ± 1.2487.5 ± 0.3877.9 ± 1.4240.3 ± 4.1681.0 ± 1.32
ZEDASFT\mathrm{ZEDA}_{\mathrm{SFT}}73.3 ± 0.3251.1 ± 0.7078.0 ± 0.5466.0 ± 0.7271.4 ± 0.4895.0 ± 0.4095.3 ± 1.6258.5 ± 1.6653.1 ± 0.5486.4 ± 1.0279.0 ± 0.6839.1 ± 1.0284.0 ± 2.10
ZEDA74.1 ± 0.1451.2 ± 0.0678.8 ± 0.7268.8 ± 2.3471.9 ± 1.8295.2 ± 0.5494.9 ± 0.6258.4 ± 1.0253.3 ± 0.6888.4 ± 0.1678.5 ± 0.4641.7 ± 3.5285.4 ± 1.92
GLM-4.7-Flash
NETSFT\mathrm{NET}_{\mathrm{SFT}}72.1 ± 0.6750.0 ± 0.0077.9 ± 3.4870.5 ± 2.8368.0 ± 4.1194.1 ± 0.2395.2 ± 0.6950.1 ± 3.6444.3 ± 1.3286.4 ± 2.1374.4 ± 2.2551.9 ± 1.3980.6 ± 2.58
NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}72.8 ± 0.3350.0 ± 0.0077.7 ± 1.8872.4 ± 3.0472.4 ± 6.2294.4 ± 0.8495.5 ± 0.6150.2 ± 1.7243.3 ± 2.7887.1 ± 3.8874.8 ± 1.5252.2 ± 1.9280.2 ± 1.53
ZEDASFT\mathrm{ZEDA}_{\mathrm{SFT}}73.1 ± 0.5152.3 ± 0.8078.8 ± 1.3771.5 ± 5.0172.8 ± 1.8795.1 ± 0.3195.3 ± 2.2749.6 ± 4.3745.0 ± 1.2187.4 ± 4.0673.2 ± 1.7152.9 ± 0.7782.9 ± 1.40
ZEDA74.2 ± 0.5552.9 ± 0.0780.1 ± 0.2473.7 ± 0.6775.1 ± 1.3494.6 ± 0.1495.4 ± 0.1851.8 ± 1.9546.0 ± 1.1986.6 ± 0.1274.0 ± 0.2253.7 ± 0.1785.2 ± 0.52

These independent runs, which include training randomness, further support ZEDA's effectiveness and its advantage over NET on both backbones.

Reviewer_WekrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose to add zero experts to existing post-trained MoEs such that during a subsequent training phase, they can learn to turn experts off while maintaining load balance across experts. This approach significantly reduces the total number of expert FLOPs required per forward pass and leads to an inference speedup of 1.2x in the empirical study. These improvements in final performance come at negligible performance degradation in the final model.

Critical strengths
  • The paper is well written and easy to follow
  • I like that the authors study a low-complexity and intuitive change thoroughly
  • The method leads to real improvements in end-to-end inference
  • The performance degradation is manageable
  • The authors provide a thorough evaluation of the method with many relevant baselines
Critical weaknesses
  • Since the models are post-trained I miss an evaluation of the benchmarks under varying pass at k. I would expect the technique to reduce the performance attainable with test-time scaling.
Questions for authors
  • The main novelty comes from the idea of zero experts and developing a recipe to make this idea work at a reasonable cost within the existing pipeline (e.g. SFT OPD). Would you say this summarizes the novelty of the contribution well?
  • How did you account for optimization hyperparameters in the study? Do they impact the main findings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Wekr

We thank the reviewer for the valuable comments and constructive questions.

W1: Pass@k Evaluation and Test-Time Scaling

We agree that pass@k evaluation is important for assessing whether ZEDA preserves the benefits of test-time scaling. We evaluated the original Qwen3-30B-A3B model, NETSFT→OPD\mathrm{NET}_{\mathrm{SFT}\rightarrow\mathrm{OPD}}, and ZEDA on AIME 24, AIME 25, and AIME 26 with k∈{1,2,4,8,16,32}k \in \{1, 2, 4, 8, 16, 32\}. The results are presented in https://postimg.cc/PC7wWVKZ.

As kk increases, the performance gap between ZEDA and the original model does not noticeably widen. These results support the preservation of test-time scaling behavior. A plausible explanation is that ZEDA retains the full pool of normal experts and learns to vary their activation across tokens, allowing more expert computation where needed. Self-distillation also encourages the adapted model to retain the original model's behavior, including along the student's own generated trajectories during OPD. Together, these properties may help preserve the benefit of sampling additional solutions.

Q1: Novelty

We would refine this characterization by emphasizing that the central novelty lies in the adaptation recipe itself, particularly the choice of zero-output experts and the group-level balancing strategy. To our knowledge, ZEDA provides the first demonstration of low-cost conversion of fully post-trained MoE models into dynamic MoEs. The specific technical contributions are: (i) introducing the Group Auxiliary Loss, which regulates utilization between normal and zero experts without explicitly enforcing uniform routing among normal experts; (ii) empirically characterizing the token-dependent computation that emerges after conversion; and (iii) systematically analyzing alternative zero-computation experts to motivate the choice of zero-output experts. These design choices make dynamic conversion effective within an established self-distillation pipeline.

Q2: Hyperparameter Selection and Sensitivity

ZEDA has two key method-specific hyperparameters, which are ww, the relative weight of the zero-expert group, and α\alpha, the coefficient of the Group Auxiliary Loss. During development, we selected the hyperparameter values based on the AIME 24, AIME 25, and MATH-500 results of models trained on mathematical data (see our response to Reviewer xZMW, W6 & Q6). The paper also presents experiments and analyses of the specific values of ww and α\alpha under the final experimental setting. We evaluate multiple values of both. Section 4.3 studies ww and shows that w=2w=2 provides a favorable trade-off between the zero-expert activation ratio rZEr_{\mathrm{ZE}} and benchmark performance. Appendix B.7 studies α\alpha, and we select α=0.1\alpha=0.1 as the smallest tested coefficient that effectively enforces the target utilization without unnecessarily strengthening the auxiliary regularization. For the remaining training hyperparameters, we primarily search learning rates on the order of 10−610^{-6} for OPD and 10−510^{-5} for SFT, selecting the best-performing value for each method. Other optimization settings follow common practice or the defaults of the slime training framework.

The main role of ww and α\alpha is to control zero-expert utilization. The reported approximately 20% inference speedup is measured at rZE≈50%r_{\mathrm{ZE}} \approx 50\%. Settings that yield a higher rZEr_{\mathrm{ZE}} may provide greater speed gains, potentially at an additional accuracy cost. However, across the alternative ww and α\alpha settings we tested, ZEDA consistently achieves the best performance among the compared methods when their expert-skipping ratios are approximately matched, from which perspective, it can be concluded that the hyperparameters do not affect the primary findings.

The reviewers have addressed my concerns. I like the paper and think it should be accepted.

I found the criticisms that 6xdb raised to be relevant and interesting; Therefore, I will wait for 6xdb to reply before deciding to raise or maintain my score.

We are grateful for the reviewer’s positive assessment, insightful suggestions, and support of our work.

For your convenience, we summarize the main additional results from our response to Reviewer 6xdb:

  • Generalization to unseen domains and agentic tasks: On four tasks not used during adaptation (BFCL V3, MMLU, GPQA, ARC-c), ZEDA comes within 0.5 points of the original model while skipping about 50% of routed-expert computation, and it outperforms all baselines. On SWE-bench Verified, ZEDA reaches 56.1 (vs. 57.4 for the original GLM-4.7-Flash), clearly outperforming NET (52.0).
  • A more recent backbone: On Qwen3.5-35B-A3B, ZEDA reaches an average accuracy of 82.0 (vs. 82.8 for the original model) with 50.9% of routed-expert computation skipped, and it remains the best among all compared methods.
  • Renormalization design: A training-free comparison shows that the two variants perform comparably. This suggests the preferred choice depends on whether the model is trained further after pruning.

We hope these results help with your assessment, and we would be glad to engage in any further discussion. Thank you again for your time and engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.