- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ZEDA, a framework for converting post-trained MoE models into dynamic ones. It combines zero-output experts, group-level balancing, and two-stage self-distillation from the original model to reduce expert computation while preserving performance. Experiments on two MoE models show over 50% less expert computation and roughly 20% higher inference throughput, with a small drop in average benchmark scores.
Critical strengths
-
The paper is clearly written and easy to follow.
-
I appreciate the honest reporting of practical speedups. In my own experiments, expert pruning has also delivered only modest gains with the highly optimized fused MoE kernels in vLLM and SGLang. The reported results are more credible to me than some of the large speedup claims in related work.
Critical weaknesses
-
I agree with using OPD to recover performance after reducing expert computation, but I have reservations about the SFT stage. Learning from a fixed set of teacher-generated responses may encourage memorization and hurt generalization. I find OPD a cleaner approach and would like to see further evaluation of generalization to unseen domains.
-
Although Qwen3-30B-A3B remains widely used, I would encourage the authors to include experiments on more recent MoE models.
-
For GLM-4.7-Flash, I would also like to see evaluations on agentic benchmarks such as SWE-bench, to assess whether the adaptation preserves the model’s agent capabilities.
-
sft / opd is both widely used post-training tech in practice, this paper used them in expert-pruning, which may reduce novelty.
Questions for authors
-
How does ZEDA compare with SlimQwen? I understand that the two are concurrent works, so I have not treated the lack of a comparison as a weakness, but I would appreciate the authors’ views on their main similarities and differences.
-
In my own experiments on dynamic expert pruning, renormalizing the remaining experts’ routing weights worked better, whereas Appendix B.6 reports the opposite result. How do the authors explain this discrepancy? In particular, could the preferred choice depend on whether the model undergoes further training after pruning?
4 comments
Response to Reviewer 6xdb (2/2)
W3: Preservation of Agentic Capabilities
We agree that agentic benchmarks provide an important additional test of capability retention. We evaluate the original GLM-4.7-Flash, , and ZEDA on SWE-bench Verified, following the GLM team's reported evaluation settings and the default OpenHands configuration. The results are shown in the table below. Compared to the original model, the ZEDA model scores slightly lower on SWE-bench Verified, significantly outperforming NET, while skipping 55.5% of routed-expert computation.
| Method | SWE-bench Verified | Avg |
|---|---|---|
| GLM-4.7-Flash | 57.4 | 0.0 |
| 52.0 | 50.0 | |
| ZEDA | 56.1 | 55.5 |
We further evaluate three models on another, slightly simpler agentic benchmark, BFCL V3. The results are shown in the table below. On these benchmarks, ZEDA continues to outperform NET, while showing almost no performance gap compared to the original model. Together, these evaluations support the preservation of the model's agent capabilities.
| Method | BFCL V3 | Avg |
|---|---|---|
| GLM-4.7-Flash | 73.7 | 0.0 |
| 71.9 | 50.0 | |
| ZEDA | 73.5 | 49.8 |
W4: Novelty
We agree that SFT and OPD are established techniques, and our novelty claim does not rest on them. We study the practically important problem of cost effectively converting a mature post trained top-K MoE into a dynamic expert model without repeating pretraining or the full post training pipeline, and to our knowledge, we are the first to successfully achieve this.
The main technical contributions are the Group Auxiliary Loss, which controls relative utilization between normal and zero experts without explicitly imposing uniform routing among normal experts, the validation of zero-expert injection as an effective conversion mechanism, an empirical characterization of the resulting computation dynamics, and a detailed analysis of alternative zero-computation experts that motivates the choice of zero-output experts.
Q1: Similarities and Differences Between ZEDA and SlimQwen
Both works seek to reduce computation while largely preserving performance, but target different stages of the model lifecycle.
ZEDA primarily targets MoE models that have undergone the complete pre-training and post-training pipeline (referred to as post-trained MoE models in our paper). These models have typically undergone SFT, RL, and OPD and are ready for deployment in inference applications. In contrast, SlimQwen focuses on compressing base models that have only undergone pre-training. As explicitly indicated by its title, “in Large MoE Model Pre-training,” both the methodological design and experimental evaluations of SlimQwen primarily focus on base models that have undergone pre-training but not subsequent post-training stages.
In terms of specific methodological design and implementation details, the two approaches share little similarity. We acknowledge that our paper lacks a discussion and citation of SlimQwen, and we will address this omission in the revised version of the paper.
Q2: Renormalization
We conduct an additional training-free experiment on Qwen3-30B-A3B. We apply only zero-expert injection, and evaluate the resulting model with and without renormalizing the remaining normal experts' routing weights. No SFT or OPD is performed. The results are shown in the table below.
| Method | Avg Acc | Avg | AIME 24 | AIME 25 | AIME 26 | GSM8K | MATH-500 | LCB v5 | LCB v6 | HumanEval+ | MBPP+ | IFBench | IFEval |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| w/ Renorm | 64.5 | 44.9 | 56.0 | 44.0 | 56.0 | 95.5 | 92.8 | 41.8 | 39.6 | 80.1 | 77.2 | 41.3 | 84.8 |
| w/o Renorm | 64.3 | 45.7 | 56.3 | 41.7 | 55.2 | 96.0 | 92.6 | 43.5 | 38.9 | 80.1 | 76.8 | 42.5 | 83.2 |
In this training-free setting, the average accuracies are close, with only a 0.2-point difference, and the non-renormalized variant shows no clear advantage. These results are consistent with the reviewer's suggestion that the preferred choice may depend on whether further training is performed. We therefore view the two observations as compatible results from different adaptation regimes, and limit our recommendation to omit renormalization to the tested adaptation setting.
Thank you for the effort you have put into the rebuttal. However, my concerns about using SFT to recover the capabilities of pruned models remain. I do not question the effectiveness or importance of SFT. Rather, I believe that such recovery training would be better integrated into pre-training as a form of “expert-pruning-aware training,” analogous to quantization-aware training (QAT). In the current pipeline, post-pruning SFT cannot cover the full diversity of the original training data, leaving concerns about overfitting to the SFT data and incomplete recovery of the model’s capabilities. I do not think the current response adequately addresses this issue.
I also continue to find the paper’s overall novelty limited. That said, I appreciate the insightful discussion of renormalization, from which I learned a lot.
Taking everything into consideration, I have decided to maintain my original score.
Thank you for considering our response. We would like to further clarify the questions you raised.
ZEDA is framed around, and specifically aims at, converting MoE models that have completed the full post-training pipeline into more efficient dynamic MoEs. Pretraining itself is outside ZEDA's intended setting. ZEDA retains all original model parameters. Since ZEDA does not perform parameter pruning, intuitions and empirical observations about overfitting in that setting may not apply directly to our approach.
Our additional experiments further show no clear signs of overfitting to the SFT data or substantial shortfalls in capability recovery on the evaluated benchmarks. On BFCL V3, MMLU, GPQA, and ARC-c, Qwen achieves 79.6 versus the original model's 80.1 average accuracy, while skipping 50% of routed-expert computation. These results provide evidence against recovery merely reflecting SFT memorization.
We respectfully view the conversion framework, Group Auxiliary Loss, and analyses of expert types, renormalization, and dynamic allocation as meaningful contributions. ZEDA's gains over NET under the same SFT→OPD stages further support the value of its design beyond established training techniques.
If any questions remain or further clarification would be helpful, please do not hesitate to let us know. We would be happy to continue the discussion. Thank you for your time and engagement.
Response to Reviewer 6xdb (1/2)
We thank the reviewer for the valuable comments and constructive suggestions.
W1: The Role of SFT and Generalization to Unseen Domains
Appendix B.3 provides evidence for the importance of the SFT stage. Table 6 (p. 18) compares SFT-only, OPD-only, and SFT→OPD under comparable total adaptation compute (the single-stage variants receive additional training steps so that their cost matches or exceeds that of the full pipeline). OPD-only achieves the lowest average accuracy, 72.9, compared with 73.6 for SFT-only and 74.2 for SFT→OPD. Our interpretation is that SFT first establishes stable zero-expert routing, whereas OPD without this initialization must simultaneously learn routing decisions and generate coherent on-policy rollouts. This makes the initial adaptation more challenging.
We agree that evaluation on unseen domains is important for assessing whether adaptation preserves capabilities beyond the training domains. We therefore additionally evaluate Qwen3-30B-A3B, both NET variants, ZEDASFT, and ZEDA on four unseen domain tasks (BFCL V3, MMLU, GPQA, and ARC-c). The results are shown in the table below. ZEDA achieves only 0.5 percentage points below the original model, while skipping approximately 50% of routed-expert computation. It outperforms all baselines. These results support capability retention in the evaluated unseen domains.
We would also clarify the intended use of ZEDA. When a deployment domain is specified, representative prompts from that domain can be included in the self-distillation corpus. Our objective is to retain capabilities while reducing computation, and it does not require substantial positive transfer to every domain excluded from adaptation. The additional results above nevertheless provide evidence that retention extends beyond the domains explicitly targeted during adaptation.
W2: Validation on a More Recent MoE Backbone
Following the reviewer's suggestion, we extend the experiments to the more recent Qwen3.5-35B-A3B, following the adaptation and evaluation settings used for Qwen3-30B-A3B. The results are shown in the table below.
ZEDA achieves 82.0 average accuracy, compared with 82.8 for the original model, while skipping 50.9% of routed-expert computation. It achieves the highest average score among all baselines. These results extend the evidence for ZEDA's effectiveness to a more recent MoE backbone.