- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Routing-and-Residual Distillation (RRD) to recover performance lost during dense-to-MoE conversion under a fixed expert partition and activation budget. RRD uses the frozen original dense model as a teacher, derives expert-selection targets from teacher neuron activations, and reduces the gap between the sparse student and the teacher through layerwise FFN-output matching, alongside cross-entropy and output-distribution distillation. Experiments on Qwen2.5-7B and Llama-2-7B at 25% and 50% FFN sparsity show higher mean zero-shot accuracy than the evaluated baselines. The authors further assess RRD through approximately compute-matched comparisons, subsequent instruction tuning, random expert partitioning, and ablation studies.
Critical strengths
1 The training procedure is clearly described, including the frozen teacher, the two student forward passes, and the parameters updated by each loss.
2 Beyond the main performance comparisons, the authors evaluate downstream fine-tuning, random expert partitioning, and teacher–student similarity, providing complementary perspectives on the method.
Critical weaknesses
1 The MoE architectures evaluated in this paper are relatively dated and differ substantially from those used in recent MoE-based large language models. The paper does not discuss the relevance of its findings to contemporary MoE architectures or the insights it may offer for their design and training.
2 The dense teacher remains frozen throughout RRD recovery. However, as the student’s expert parameters and routing policy evolve, it is unclear whether the fixed teacher targets remain appropriate for the student’s current sparse computation. In particular, could routing distillation and FFN-output matching constrain further improvements in task performance during the middle or later stages of training? The existing experiments do not directly address this question.
3 Have the authors considered periodically updating the teacher, for example by reassembling the student’s MoE expert weights into a dense model after a certain number of training steps and using the resulting checkpoint to refresh the teacher? An analysis of the effects of fixed-teacher supervision across training stages, together with a discussion of the feasibility of this approach, would be useful. Comparing fixed and periodically refreshed teachers under comparable training compute budgets would help establish the scope of the current design.
4 The evaluated configurations are only moderately sparse: they still activate 75% and 50% of the FFN partitions per token, respectively. It therefore remains unclear whether RRD is effective at substantially lower expert activation ratios. Experiments at higher sparsity levels would help establish the method’s applicability and practical value.
5 Routing distillation and residual distillation are introduced together, but the paper does not evaluate the effect of adding each objective individually. Consequently, their respective contributions remain unclear, and the possibility that one component accounts for most of the improvement cannot be ruled out.
6 The ablation studies use a substantially smaller training budget than the main experiments. The core ablations use approximately 4M tokens, whereas the main comparisons use approximately 40M. An objective that helps early in training may not provide the same benefit after longer training, so the key conclusions should be verified under the main training budget.
Questions for authors
see Weaknesses
3 comments
W3. "Have the authors considered periodically updating the teacher ..."
(Revised manuscript: Appendix D.3, Tab. 15, p. 21, lines 1097–1116)
Thank you for this suggestion. It fits our setting well, because the experts are a partition of the original FFN and can be merged back into a dense model at any point in training. We examined the condition it depends on in the revised version.
Replacing the teacher helps only if the reassembled model is a stronger teacher than the original dense model. We therefore measured this first. We merged all experts of each trained MoE back into a dense model and evaluated it.
Table R3. Sparse execution vs. reassembled dense model
| Qwen2.5-7B E8S2A2 | C4, sparse (50%) | C4, reassembled (100%) | WT2, sparse (50%) | WT2, reassembled (100%) |
|---|---|---|---|---|
| Original dense (teacher) | – | 11.81 | – | 7.49 |
| CMoE w/o CPT | 42.23 | 11.81$^\ast$ | 41.05 | 7.49$^\ast$ |
| CMoE + CE | 18.63 | 17.57 | 16.61 | 11.98 |
| CMoE + (CE + Logit-KD) | 17.43 | 17.05 | 13.76 | 11.43 |
| CMoE + LoRA | 20.60 | 14.90 | 17.90 | 10.66 |
| CMoE + RRD | 16.80 | 16.46 | 12.99 | 10.97 |
Right after conversion the FFN is only partitioned, so merging all experts returns the original dense model.
- The original dense model remains the stronger teacher. After 40M tokens, every reassembled model has higher perplexity than the original teacher (for RRD, 16.46 vs. 11.81 on C4 and 10.97 vs. 7.49 on WT2). Replacing the teacher at this point would lower the quality of the targets.
- Reassembly still recovers part of the gap. The reassembled RRD model is better than its own sparse execution (C4 16.80 → 16.46, WT2 12.99 → 10.97), so the inactive experts keep useful information.
For this reason we keep the original teacher fixed in this work. The W2 results agree with this choice, since the fixed targets remain useful through the second half of training. We added Table R3 to the Appendix.
W4. "The evaluated configurations are only moderately sparse ..."
(Revised manuscript: Appendix D.4, Tab. 17-18, p. 22-23, lines 1173–1199)
We added a 75% sparsity setting, where each token activates only 25% of the FFN (Table R1). The table below collects the gain of RRD over CE + Logit-KD fine-tuning under the same setup at all three sparsity levels.
Table R4. Average zero-shot accuracy over five benchmarks
| Qwen2.5-7B | E8S3A3 (Sp. 25%) | E8S2A2 (Sp. 50%) | E32S1A7 (Sp. 75%) |
|---|---|---|---|
| CMoE + (CE + Logit-KD) | 65.77 | 57.65 | 47.67 |
| CMoE + RRD | 67.54 | 59.71 | 50.76 |
| Gain | +1.77 | +2.06 | +3.09 |
The gain of RRD over CE + Logit-KD does not shrink as sparsity increases (+1.77, +2.06, and +3.09 pp).
W5. "Routing distillation and residual distillation are introduced together, but the paper does not evaluate the effect of adding each objective individually. ..."
(Revised manuscript: Sec. 4.6, Fig. 2-3, p. 8-9, lines 421–468)
We added both single-objective variants to Fig. 2.
Table R5. Effect of adding each distillation objective at the 40M budget (unweighted top-K, as in the paper)
| Qwen2.5-7B E8S2A2 | Avg. Acc. | C4 PPL ↓ | WT2 PPL ↓ |
|---|---|---|---|
| CE only | 54.78 | 19.00 | 17.30 |
| CE + Logit-KD | 57.49 | 17.92 | 14.28 |
| CE + Logit-KD + RouDistill | 59.47 | 16.97 | 13.32 |
| CE + Logit-KD + ResDistill | 57.44 | 17.71 | 13.96 |
| CE + Logit-KD + RouDistill + ResDistill (RRD) | 59.71 | 16.80 | 12.99 |
Routing distillation (RouDistill) accounts for most of the accuracy gain (+1.98 pp of the +2.22 pp over CE + Logit-KD). Residual distillation (ResDistill) adds a smaller but consistent gain.
- On top of RouDistill, it improves all three metrics (C4 PPL 16.97 → 16.80, WT2 PPL 13.32 → 12.99, accuracy +0.24 pp).
- Alone, it improves PPL but not accuracy. In that row, however, the router is frozen, because the output losses cannot update it under unweighted top-K. In a control where the router is trained by the output losses, adding ResDistill gives +1.27 pp (57.65 → 58.92; Appendix D.3, Tab. 13).
RRD, which uses both, is the best on all three metrics. We revised the text to state the relative contributions explicitly.
W6. "The ablation studies use a substantially smaller training budget than the main experiments. ..."
(Revised manuscript: Sec. 4.6, Fig. 2-3, p. 8-9, lines 421–468)
We re-ran all ablations in Fig. 2 and Fig. 3 at the main 40M budget (Table R5), and the key conclusions hold. RRD is the best variant on all three metrics. It is 2.22 pp above CE + Logit-KD (57.49 → 59.71) and 4.93 pp above CE only (54.78 → 59.71). Both RouDistill and ResDistill contribute at this budget (W5).
We are following up on our response to your review, in case it was missed during this busy period.
We appreciate your constructive comments, especially the questions about the fixed teacher, which led to analyses we found valuable.
In our response, we did our best to address each concern with both quantitative and qualitative evidence, and the revised manuscript has been updated accordingly.
We understand you are busy, but if any of your concerns remain, we would be grateful if you could let us know so we can address them during the discussion period.
If you feel they have been resolved, we would appreciate it if you would kindly consider updating your score.
Thank you for the careful review and the constructive comments. In addressing them, we added the following to the revised version.
Unless otherwise noted, results use Qwen2.5-7B E8S2A2 (50% FFN sparsity) and 40M C4 tokens.
W1. "... The MoE architectures evaluated in this paper are relatively dated and differ substantially from those used in recent MoE-based large language models. ..."
(Revised manuscript: Appendix D.4, Tab. 17–18, p. 22–23, lines 1173–1199)
Recent MoE LLMs use many fine-grained experts together with shared experts and activate a small fraction of them per token. To move toward this design within dense-to-MoE conversion, we converted Qwen2.5-7B to E32S1A7, which splits each FFN into 32 experts and activates one shared and seven routed experts per token (75% FFN sparsity).
Table R1. Fine-grained topology with a shared expert
RRD remains the best of the three methods, 3.09 pp above CE + Logit-KD and 7.07 pp above LoRA, with the lowest PPL on both datasets. The two RRD objectives do not depend on expert granularity. The routing target is defined per routed group, and the residual is defined on the layer output.
W2. "... it is unclear whether the fixed teacher targets remain appropriate for the student's current sparse computation. ..."
(Revised manuscript: Sec. 4.8, Fig. 4, p. 10, lines 506–514; Appendix D.3, Tab. 14, Fig. 6, p. 20–21, lines 1059–1095)
We tested this directly. We split the 40M budget into two 20M stages and changed the objective at the midpoint.
Table R2. Objective switching at the midpoint (Qwen2.5-7B E8S2A2)
Because some second stages have no routing distillation (RouDistill), all runs in this table use the softmax-weighted variant in which the router is trained by all active losses (revised Sec. 4.7). This is why RRD → RRD is 59.01 here, not the 59.71 of the main RRD.
So within this budget we find no sign that routing distillation or FFN-output matching limits task performance in the later stage. This is consistent with the remaining gap. After training, the student (C4 PPL 16.80) is still far from the dense teacher (11.81), so the fixed teacher is still an informative target.