acceptodds
Under review as a conference paper at ICLR 2027

RRD: Routing-and-Residual Distillation for Recovery after Dense-to-MoE Conversion

Abstract

Mixture-of-experts (MoE) models reduce per-token feed-forward network (FFN) computation by activating only a subset of experts. Dense-to-MoE conversion reuses pretrained dense weights, but sparse top- execution can create a gap between the converted student and its dense teacher. We formulate recovery under a fixed activation budget as two linked objectives: selecting useful experts and matching layerwise FFN outputs. Cross-entropy (CE) and logit-based knowledge distillation (logit-KD) supervise next-token predictions but do not directly constrain either objective. We propose Routing-and-Residual Distillation (RRD), which derives top- routing targets from teacher neuron activations and matches the realized sparse FFN output to the corresponding dense teacher FFN output. RRD combines these internal objectives with CE and logit-KD while retaining the converted expert partition and fixed activation budget. Across Qwen2.5-7B and Llama-2-7B, RRD achieves the highest mean zero-shot accuracy among the evaluated methods at both 25% and 50% FFN sparsity. On Qwen2.5-7B at 25% sparsity, RRD reaches 68.40% mean zero-shot accuracy after training on about 40M C4 tokens, outperforming the strongest same-topology baseline by 3.13 percentage points. At 50% sparsity, it outperforms the strongest compute-matched baseline by 2.47 percentage points under an approximately matched training-FLOP budget that includes the frozen teacher's forward pass. These results support direct supervision of expert selection and layerwise FFN outputs during sparse recovery. Code is available at https://anonymous.4open.science/r/routing_and_residual_distillation-7226/

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Knowledge Distillation, Routing distillation, Representation distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gsHhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Routing-and-Residual Distillation (RRD) to recover performance lost during dense-to-MoE conversion under a fixed expert partition and activation budget. RRD uses the frozen original dense model as a teacher, derives expert-selection targets from teacher neuron activations, and reduces the gap between the sparse student and the teacher through layerwise FFN-output matching, alongside cross-entropy and output-distribution distillation. Experiments on Qwen2.5-7B and Llama-2-7B at 25% and 50% FFN sparsity show higher mean zero-shot accuracy than the evaluated baselines. The authors further assess RRD through approximately compute-matched comparisons, subsequent instruction tuning, random expert partitioning, and ablation studies.

Critical strengths

1 The training procedure is clearly described, including the frozen teacher, the two student forward passes, and the parameters updated by each loss.

2 Beyond the main performance comparisons, the authors evaluate downstream fine-tuning, random expert partitioning, and teacher–student similarity, providing complementary perspectives on the method.

Critical weaknesses

1 The MoE architectures evaluated in this paper are relatively dated and differ substantially from those used in recent MoE-based large language models. The paper does not discuss the relevance of its findings to contemporary MoE architectures or the insights it may offer for their design and training.

2 The dense teacher remains frozen throughout RRD recovery. However, as the student’s expert parameters and routing policy evolve, it is unclear whether the fixed teacher targets remain appropriate for the student’s current sparse computation. In particular, could routing distillation and FFN-output matching constrain further improvements in task performance during the middle or later stages of training? The existing experiments do not directly address this question.

3 Have the authors considered periodically updating the teacher, for example by reassembling the student’s MoE expert weights into a dense model after a certain number of training steps and using the resulting checkpoint to refresh the teacher? An analysis of the effects of fixed-teacher supervision across training stages, together with a discussion of the feasibility of this approach, would be useful. Comparing fixed and periodically refreshed teachers under comparable training compute budgets would help establish the scope of the current design.

4 The evaluated configurations are only moderately sparse: they still activate 75% and 50% of the FFN partitions per token, respectively. It therefore remains unclear whether RRD is effective at substantially lower expert activation ratios. Experiments at higher sparsity levels would help establish the method’s applicability and practical value.

5 Routing distillation and residual distillation are introduced together, but the paper does not evaluate the effect of adding each objective individually. Consequently, their respective contributions remain unclear, and the possibility that one component accounts for most of the improvement cannot be ruled out.

6 The ablation studies use a substantially smaller training budget than the main experiments. The core ablations use approximately 4M tokens, whereas the main comparisons use approximately 40M. An objective that helps early in training may not provide the same benefit after longer training, so the key conclusions should be verified under the main training budget.

Questions for authors

see Weaknesses

3 comments

Thank you for the careful review and the constructive comments. In addressing them, we added the following to the revised version.

  • Experiments at 75% FFN sparsity with a finer-grained topology (E32S1A7), and a discussion of how RRD relates to recent MoE designs (W1, W4)
  • An analysis of the fixed teacher, with objective-switching experiments across training stages (W2) and an evaluation of reassembled dense models as candidate refreshed teachers (W3)
  • Ablations that add routing distillation (RouDistill) and residual distillation (ResDistill) individually, at the main 40M budget (W5, W6)

Unless otherwise noted, results use Qwen2.5-7B E8S2A2 (50% FFN sparsity) and 40M C4 tokens.

W1. "... The MoE architectures evaluated in this paper are relatively dated and differ substantially from those used in recent MoE-based large language models. ..."
(Revised manuscript: Appendix D.4, Tab. 17–18, p. 22–23, lines 1173–1199)

Recent MoE LLMs use many fine-grained experts together with shared experts and activate a small fraction of them per token. To move toward this design within dense-to-MoE conversion, we converted Qwen2.5-7B to E32S1A7, which splits each FFN into 32 experts and activates one shared and seven routed experts per token (75% FFN sparsity).

Table R1. Fine-grained topology with a shared expert

Qwen2.5-7B E32S1A7 (75% sparsity)PIQAWinoGrandeARC-eARC-cHellaSwagAvg. Acc.C4 PPL ↓WT2 PPL ↓
CE + Logit-KD67.1955.7244.8226.1944.4047.6724.8822.87
LoRA60.5053.8341.7124.4038.0143.6936.9833.71
RRD69.3155.3349.1230.2949.7550.7622.5719.67

RRD remains the best of the three methods, 3.09 pp above CE + Logit-KD and 7.07 pp above LoRA, with the lowest PPL on both datasets. The two RRD objectives do not depend on expert granularity. The routing target is defined per routed group, and the residual is defined on the layer output.

W2. "... it is unclear whether the fixed teacher targets remain appropriate for the student's current sparse computation. ..."
(Revised manuscript: Sec. 4.8, Fig. 4, p. 10, lines 506–514; Appendix D.3, Tab. 14, Fig. 6, p. 20–21, lines 1059–1095)

We tested this directly. We split the 40M budget into two 20M stages and changed the objective at the midpoint.

  • If the fixed-teacher targets constrain the student late in training, removing them in the second stage should help.
  • If they are still useful, adding them in the second stage should help.

Table R2. Objective switching at the midpoint (Qwen2.5-7B E8S2A2)

First 20M → second 20MChange at midpointAvg. Acc.C4 PPL ↓WT2 PPL ↓
RRD → RRDNone59.0117.4113.40
RRD → CE + RouDistill + ResDistill− Logit-KD58.0217.8214.13
RRD → CE + Logit-KD− RouDistill, ResDistill56.8717.8914.31
RRD → CE− all three56.4918.8016.26
CE → CENone55.7518.6316.61
CE → CE + Logit-KD+ Logit-KD56.8217.7214.08
CE → CE + RouDistill + ResDistill+ RouDistill, ResDistill57.0018.5014.60
CE → RRD+ all three57.5317.9713.94

Because some second stages have no routing distillation (RouDistill), all runs in this table use the softmax-weighted variant in which the router is trained by all active losses (revised Sec. 4.7). This is why RRD → RRD is 59.01 here, not the 59.71 of the main RRD.

  • Removing teacher supervision late always hurts. Compared with RRD → RRD, dropping Logit-KD costs 0.99 pp, dropping RouDistill and ResDistill costs 2.14 pp, and dropping all three costs 2.52 pp, with higher PPL in every case.
  • Adding teacher supervision late always helps. CE → RRD is 1.78 pp above CE → CE, and each partial addition also helps (+1.07 pp, +1.25 pp).
  • Early supervision also matters. CE → RRD (57.53) remains below RRD → RRD (59.01).

So within this budget we find no sign that routing distillation or FFN-output matching limits task performance in the later stage. This is consistent with the remaining gap. After training, the student (C4 PPL 16.80) is still far from the dense teacher (11.81), so the fixed teacher is still an informative target.

W3. "Have the authors considered periodically updating the teacher ..."
(Revised manuscript: Appendix D.3, Tab. 15, p. 21, lines 1097–1116)

Thank you for this suggestion. It fits our setting well, because the experts are a partition of the original FFN and can be merged back into a dense model at any point in training. We examined the condition it depends on in the revised version.

Replacing the teacher helps only if the reassembled model is a stronger teacher than the original dense model. We therefore measured this first. We merged all experts of each trained MoE back into a dense model and evaluated it.

Table R3. Sparse execution vs. reassembled dense model

Qwen2.5-7B E8S2A2C4, sparse (50%)C4, reassembled (100%)WT2, sparse (50%)WT2, reassembled (100%)
Original dense (teacher)–11.81–7.49
CMoE w/o CPT42.2311.81$^\ast$41.057.49$^\ast$
CMoE + CE18.6317.5716.6111.98
CMoE + (CE + Logit-KD)17.4317.0513.7611.43
CMoE + LoRA20.6014.9017.9010.66
CMoE + RRD16.8016.4612.9910.97

∗^\ast Right after conversion the FFN is only partitioned, so merging all experts returns the original dense model.

  • The original dense model remains the stronger teacher. After 40M tokens, every reassembled model has higher perplexity than the original teacher (for RRD, 16.46 vs. 11.81 on C4 and 10.97 vs. 7.49 on WT2). Replacing the teacher at this point would lower the quality of the targets.
  • Reassembly still recovers part of the gap. The reassembled RRD model is better than its own sparse execution (C4 16.80 → 16.46, WT2 12.99 → 10.97), so the inactive experts keep useful information.

For this reason we keep the original teacher fixed in this work. The W2 results agree with this choice, since the fixed targets remain useful through the second half of training. We added Table R3 to the Appendix.

W4. "The evaluated configurations are only moderately sparse ..."
(Revised manuscript: Appendix D.4, Tab. 17-18, p. 22-23, lines 1173–1199)

We added a 75% sparsity setting, where each token activates only 25% of the FFN (Table R1). The table below collects the gain of RRD over CE + Logit-KD fine-tuning under the same setup at all three sparsity levels.

Table R4. Average zero-shot accuracy over five benchmarks

Qwen2.5-7BE8S3A3 (Sp. 25%)E8S2A2 (Sp. 50%)E32S1A7 (Sp. 75%)
CMoE + (CE + Logit-KD)65.7757.6547.67
CMoE + RRD67.5459.7150.76
Gain+1.77+2.06+3.09

The gain of RRD over CE + Logit-KD does not shrink as sparsity increases (+1.77, +2.06, and +3.09 pp).

W5. "Routing distillation and residual distillation are introduced together, but the paper does not evaluate the effect of adding each objective individually. ..."
(Revised manuscript: Sec. 4.6, Fig. 2-3, p. 8-9, lines 421–468)

We added both single-objective variants to Fig. 2.

Table R5. Effect of adding each distillation objective at the 40M budget (unweighted top-K, as in the paper)

Qwen2.5-7B E8S2A2Avg. Acc.C4 PPL ↓WT2 PPL ↓
CE only54.7819.0017.30
CE + Logit-KD57.4917.9214.28
CE + Logit-KD + RouDistill59.4716.9713.32
CE + Logit-KD + ResDistill57.4417.7113.96
CE + Logit-KD + RouDistill + ResDistill (RRD)59.7116.8012.99

Routing distillation (RouDistill) accounts for most of the accuracy gain (+1.98 pp of the +2.22 pp over CE + Logit-KD). Residual distillation (ResDistill) adds a smaller but consistent gain.

  • On top of RouDistill, it improves all three metrics (C4 PPL 16.97 → 16.80, WT2 PPL 13.32 → 12.99, accuracy +0.24 pp).
  • Alone, it improves PPL but not accuracy. In that row, however, the router is frozen, because the output losses cannot update it under unweighted top-K. In a control where the router is trained by the output losses, adding ResDistill gives +1.27 pp (57.65 → 58.92; Appendix D.3, Tab. 13).

RRD, which uses both, is the best on all three metrics. We revised the text to state the relative contributions explicitly.

W6. "The ablation studies use a substantially smaller training budget than the main experiments. ..."
(Revised manuscript: Sec. 4.6, Fig. 2-3, p. 8-9, lines 421–468)

We re-ran all ablations in Fig. 2 and Fig. 3 at the main 40M budget (Table R5), and the key conclusions hold. RRD is the best variant on all three metrics. It is 2.22 pp above CE + Logit-KD (57.49 → 59.71) and 4.93 pp above CE only (54.78 → 59.71). Both RouDistill and ResDistill contribute at this budget (W5).

We are following up on our response to your review, in case it was missed during this busy period.
We appreciate your constructive comments, especially the questions about the fixed teacher, which led to analyses we found valuable.

In our response, we did our best to address each concern with both quantitative and qualitative evidence, and the revised manuscript has been updated accordingly.

We understand you are busy, but if any of your concerns remain, we would be grateful if you could let us know so we can address them during the discussion period.
If you feel they have been resolved, we would appreciate it if you would kindly consider updating your score.

Reviewer_n8RrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies recovery after converting a pretrained dense language model into a sparsely activated MoE, while retaining the expert partition and per-token activation budget. Routing-and-Residual Distillation (RRD) combines two internal supervision objectives with cross-entropy and logit distillation. Routing distillation constructs token-specific top-K expert targets from the absolute activation mass of teacher neurons assigned to each expert. Residual distillation matches the student’s realized sparse FFN outputs to the corresponding dense teacher outputs. Expert parameters are trained under student-selected routing, while a separate teacher-forced student pass supplies router supervision. Experiments cover Qwen2.5-7B and Llama-2-7B at 25% and 50% FFN sparsity, using approximately 42M C4 token positions. RRD achieves the highest five-task mean accuracy among the evaluated methods in all four settings. Additional experiments examine approximate training-FLOP matching, subsequent teacher-free supervised fine-tuning, a random expert partition, and shorter loss ablations.

Critical strengths
  1. Consistent empirical improvements across several relevant settings. Tables 1 and 2 show accuracy gains across both backbones and both sparsity levels. The method also shows consistent improvement in the reported FLOP-matched comparison. These results indicate that the recovery recipe is useful within the evaluated regime.
  2. The method is applicable beyond one particular checkpoint or evaluation stage. Applying RRD to the random-split LLaMA-MoE configuration improves mean accuracy. The common teacher-free SFT procedure also preserves advantages in five-task accuracy, MMLU, and GSM8K.
Critical weaknesses
  1. Limited novelty. The two major components are already studied in prior works. MoEfication already trains an activation-supervised router on dense-model representations. LaDiMo already uses layerwise distillation for dense-to-MoE conversion. Intermediate-output matching plus logit KD is standard (FitNets, Patient KD).
  2. The specific designs of RRD include a) computing L_router on student states from a second, teacher-forced pass; b) the hard split of gradients (output losses to experts, L_router to the router); c) computing the layerwise residual end-to-end on the student’s own state. But none of these choices is ablated against its obvious alternative, such as ordinary-pass states, a gated router, or layerwise matching on a shared input. LaDiMo is not compared. It is also untested whether summed absolute activation is a good routing target, as Fig. 3b measures agreement with the same oracle L_router is trained on.
  3. RRD updates all shared and routed expert weights directly. The same-topology baseline CMoE + LoRA, instead, trains rank-8 LoRA adapters on attention and expert projections, plus its router and scaling parameters. The gains over CMoE + LoRA mix the effect of the RRD losses with a large difference in trainable capacity. The compute-matched Table 3 also does not include CMoE + LoRA.
  4. No ablations of the contributions of the two proposed losses. In the controlled ablation in Fig. 2, there is no variant that adds only L_router or only L_residual to CE + Logit-KD. As I read Secs. 2 and 3.4, every variant without L_router has a router that receives no gradient at all, because selected experts are summed without gate weights. In Fig. 3b, the CE-only and Logit-KD-only variants sit near the 2/6 chance rate, which fits that reading. If so, the ablation compares routing distillation with a frozen router, not with a router trained from the output losses, which is the natural alternative. And the controlled ablation only runs at 4M tokens, which is a tenth of the main budget.
Questions for authors
  1. In the Fig. 2/3 variants without L_router, does the router get any training signal, or does it stay at its CMoE initialization? If it is frozen, please add a control in which the router is trained from the output losses under the same expert fine-tuning. This could use softmax- or scale-weighted top-K, or a straight-through estimator.
  2. Is it possible to get the CMoE + full expert fine-tuning with CE, and with CE + Logit-KD, at E8S3A3 and E8S2A2? These runs should use RRD’s trainable parameters, optimizer, learning rate, and 40M-token budget. Please also add both this control and CMoE + LoRA to the compute-matched Table 3. This separates the objective from the difference between full fine-tuning and LoRA.
  3. Why is L_router computed on a second, teacher-forced pass rather than on the student states already produced in step 6 of Algorithm 1? At inference, upstream layers follow the router’s own choices. With hit fractions of roughly 0.5–0.75 in Fig. 3b, those states differ from the teacher-forced ones. Two additions would show how much remains to be gained from routing versus expert capacity: an ablation using ordinary-pass states, and the final student’s accuracy when run with the teacher-derived S* as an oracle-routing upper bound. Please also state the router architecture. Also, what accounts for the 0.07–0.11B extra parameters of CMoE + RRD relative to CMoE + LoRA in Table 6?
  4. Could you provide ablations with CE + Logit-KD + L_router and CE + Logit-KD + L_residual, under 40M budget to show the benefits of the proposed losses?
7 comments

Thank you for the detailed and concrete review. We addressed every comment and question, and the revised version discusses them in detail. The main additions are as follows.

  • A clearer positioning against MoEfication and LaDiMo, with a MoEfication-style baseline and a LaDiMo-style comparison (W1)
  • Design ablations on ordinary-pass vs. teacher-forced router states, shared-input residual matching, and a gated router (W2, Q1, Q3)
  • Full fine-tuning baselines with RRD's training setup, and CMoE + LoRA in the compute-matched Table 3 (W3, Q2)
  • An oracle-routing upper bound and the router architecture (Q3)
  • Ablations that add routing distillation (RouDistill) and residual distillation (ResDistill) individually at the 40M budget (W4, Q4)

Unless otherwise noted, results use Qwen2.5-7B E8S2A2 (50% FFN sparsity) and 40M C4 tokens.

W1. "Limited novelty. The two major components are already studied in prior works. ..."
(Revised manuscript: Appendix D.2, Tab. 10–11, p. 18–19, lines 964–1016)

We agree that these are the closest prior works. We do not claim activation-supervised routing or layerwise matching as new ideas in themselves. RRD combines teacher-derived expert selection with residual matching along the student's own computation path for dense-to-MoE recovery. Its contribution should be assessed through this formulation and its controlled evidence, rather than through the novelty of the individual KD ingredients.

We compared against MoEfication and LaDiMo.

W1a. MoEfication-style comparison. MoEfication trains the router once, offline, on dense-model representations, and then keeps it fixed. It can optionally fine-tune only the down projections of the experts. In RRD, the router and the experts are trained together, and all projection weights of the experts are trained.

We applied MoEfication to Qwen2.5-7B. We follow MoEfication’s expert width of 32 neurons. To match the 50% FFN sparsity of our main experiments, we partition each Qwen2.5-7B FFN into 592 experts and activate 296 experts per token.

We compared the following four settings.

  • (1) MoEfication. The router is trained offline on activation labels and then frozen. Only the down projections are tuned with CE.
  • (2) Frozen-router RRD. The router is trained offline on the RRD routing target and then frozen. Only the down projections are tuned with the RRD expert losses (CE + Logit-KD + ResDistill).
  • (3) Frozen-router RRD w/o ResDistill. Same as (2), but without residual distillation.
  • (4) RRD. The router keeps learning from routing distillation during recovery, and all expert projections are trained.

Table R1. Recovery on the MoEfication partition (Qwen2.5-7B, 592 experts, top-296)

Qwen2.5-7B E592S0A296Router during recoveryTrainable expert projectionsExpert objectiveAvg. Acc.C4 PPL ↓WT2 PPL ↓
MoEficationFrozen, activation-supervisedDownCE64.5914.1211.08
Frozen-router RRDFrozen, routing-distilledDownCE + Logit-KD + ResDistill63.5015.3210.66
Frozen-router RRD w/o ResDistillSame frozen routerDownCE + Logit-KD63.1014.7410.34
RRDUpdated by RouDistillGate / up / downCE + Logit-KD + ResDistill65.3015.2510.69
  • RRD has the best accuracy, and MoEfication has the best C4 PPL. RRD is 0.71 pp above MoEfication and has lower WT2 PPL, but higher C4 PPL (15.25 vs. 14.12).
  • With a frozen router and down-only training, residual distillation (ResDistill) raises accuracy (+0.40 pp) but not PPL (rows 2–3). Its benefit is clearer when all expert projections are trained (Tables R5 and R12).

This partition is much finer than the E8 topologies of our main experiments, which follow the CMoE setting, so these numbers should be compared within this table. MoEfication also uses a larger token budget, about 46.64M tokens across graph construction, offline router training, and expert calibration.

W1b. LaDiMo-style comparison. We reproduced the LaDiMo setting on Llama-2-7B (only the last 12 layers converted). Topology (E8S2A2), router architecture, top-2 routing, and the 40M budget are matched to RRD.

Table R2. Comparison in the LaDiMo setting

Llama-2-7BConverted layers (topology)Active params (B)Avg. Acc.C4 PPL ↓WT2 PPL ↓
CMoE + RRDAll layers (E8S3A3)5.7963.748.5210.62
CMoE + RRDAll layers (E8S2A2)4.7158.4510.0813.24
LaDiMo-style local KDLast 12 (E8S2A2)5.9865.069.3512.67
CMoE + RRDLast 12 (E8S2A2)5.9866.478.1910.41

In the same conversion setting, RRD is 1.41 pp above LaDiMo-style local KD, with C4 and WT2 PPL lower by 1.16 and 2.26, respectively. RRD with all layers converted (E8S3A3) uses fewer active parameters than LaDiMo-style local KD and still has lower C4 and WT2 PPL.

W2. "... none of these choices is ablated against its obvious alternative ..."

Following your suggestions, we ran the ablations below.

  • (a) Ordinary-pass states for the router loss (Q3a)
  • (b) Gated router that also receives output-loss gradients (Q1)
  • cc Layerwise matching on a shared input (below)
  • LaDiMo comparison (W1)
  • Whether summed absolute activation is a good routing target (Q3b)

cc Layerwise matching on a shared input
(Revised manuscript: Sec. 4.9, Fig. 5, p. 10, lines 516–525; Appendix D.3, Tab. 16, Fig. 7, p. 21–22, lines 1117–1170)

Shared-input matching is on par with RRD on the training distribution (C4), but RRD generalizes better to unseen text (WikiText-2). Adding upstream gradients to shared-input matching clearly hurts.

We changed only the residual term and kept everything else fixed. Teacher-input and Student-input give the MoE layer and the dense target the same input, and stop the residual gradient at that layer. Student-input + upstream grad also lets the residual gradient flow into earlier layers, as RRD does.

Table R3. Input path of residual distillation

Qwen2.5-7B E8S2A2MoE inputDense target inputAvg. Acc.C4 PPL ↓WT2 PPL ↓
RRD (own paths)StudentTeacher59.7116.8012.99
Teacher-inputTeacherTeacher60.1516.8213.30
Student-inputStudentStudent60.2116.8313.22
Student-input + upstream gradStudentStudent55.2518.3214.95
  • Training distribution (C4). Teacher-input and Student-input match RRD within 0.03 PPL, and their average accuracy is slightly higher (+0.4 to +0.5 pp).
  • Unseen distribution (WT2). RRD has the lowest perplexity. Table R4 compares the models window by window. Against Teacher-input and Student-input, RRD is lower on 85–92% of the windows, and both 95% intervals exclude zero.
  • Upstream gradients. With a shared student input, upstream gradients degrade every metric (Avg. Acc. −4.5 pp, WT2 PPL +15%). In RRD the upstream gradient is paired with a target from the teacher’s own path, and we attribute the difference to this pairing.

Table R4. Per-window WT2 perplexity relative to RRD (265 windows, 10,000 bootstrap resamples)

Variant relative to RRDHigher WT2 PPL95% intervalWindows with lower PPL for RRD
Teacher-input2.35%[2.09%, 2.65%]245/265
Student-input1.79%[1.52%, 2.08%]226/265
Student-input + upstream grad15.04%[14.24%, 16.16%]264/265
  • Own-path matching adds no forward computation. Both of its targets are already produced by the teacher pass and the student pass. In measured FLOPs per step, Teacher-input is 3.3% cheaper than RRD, and Student-input and Student-input + upstream grad are 6.7% and 10.1% more expensive.

In conclusion, we keep own-path matching as the default. It gives the best perplexity at a comparable cost.

W3. "... The gains over CMoE + LoRA mix the effect of the RRD losses with a large difference in trainable capacity. The compute-matched Table 3 also does not include CMoE + LoRA."

We agree. We added full fine-tuning baselines with the same trainable parameters and training setup as RRD (Q2a), and added them and CMoE + LoRA to the compute-matched comparison (Q2b). RRD is the best in both.

W4. "No ablations of the contributions of the two proposed losses. ..."

In the original Fig. 2, the router of every variant without Lrouter\mathcal{L}_{\mathrm{router}}​ received no gradient and stayed at its initialization. Q1 reports controls with a router trained by the output losses, and Q4 reports each loss added individually at 40M.

Q1. "... add a control in which the router is trained from the output losses under the same expert fine-tuning." (Revised: Sec. 4.7, Tab. 6, p. 9, lines 471–485; App. D.3, Tab. 12–13, p. 19–20, lines 1019–1057)

Yes, the router stayed at its initialization in those variants. As suggested, we added controls with a softmax-weighted top-K sum, so that the output losses train the router under the same expert fine-tuning. We also tested sending the output-loss gradients to the router together with routing distillation (RouDistill).

Table R5. Comparison by router training signal

ObjectivesAggregationRouter gradientsAvg. Acc.C4 PPL ↓WT2 PPL ↓
CEUnweightedFrozen54.7819.0017.30
CEWeightedCE55.7518.6316.61
CE + Logit-KDUnweightedFrozen57.4917.9214.28
CE + Logit-KDWeightedCE + Logit-KD57.6517.4313.76
CE + Logit-KD + ResDistillUnweightedFrozen57.4417.7113.96
CE + Logit-KD + ResDistillWeightedCE + Logit-KD + ResDistill58.9217.2413.30
CE + Logit-KD + RouDistillWeightedAll enabled losses58.4417.5713.86
CE + Logit-KD + RouDistillUnweightedRouDistill only$^\ast$59.4716.9713.32
RRDWeightedAll losses59.0117.4113.40
RRDUnweightedRouDistill only$^\ast$59.7116.8012.99

∗^\ast The setting used in the paper. "Unweighted" is a plain top-K sum, and "Weighted" is a softmax-weighted top-K sum.

  • Output losses do train the router, but less effectively than RouDistill. A router trained by the output losses beats a frozen one in every pair, yet the best such variant (58.92) is still below RRD (59.71).
  • Adding the output-loss gradients to the router does not help. With RouDistill present, the weighted variant that also sends the output losses to the router is lower (59.47 → 58.44, 59.71 → 59.01).

In summary, both objectives contribute, and RouDistill contributes more than ResDistill. The current design, which uses both and trains the router with RouDistill only, gives the best result.

In addition, when the router is trained from the output losses (CE + Logit-KD), the top-2 hit fraction, the routing-fidelity measure of Fig. 3, stays at chance. It is 0.334 on C4, where chance is 2/6 = 0.333, compared with 0.638 for RRD. The output losses adjust the router enough to lower the loss, but they do not move it toward the teacher's expert selection. Only RouDistill does.

Q2a. "Is it possible to get the CMoE + full expert fine-tuning with CE, and with CE + Logit-KD, at E8S3A3 and E8S2A2? ..." (Revised: Sec. 4.7, Tab. 6, p. 9, lines 470–485; App. C, p. 17, lines 903–907; App. D.3, Tab. 12, p. 20, lines 1030–1037)

Yes. We added both, with the same trainable parameters, optimizer, learning rate, and 40M budget as RRD. Their router is trained jointly by the output losses.

Table R6. Recovery at the main token budget

Qwen2.5-7B E8S3A3Avg. Acc.C4 PPL ↓WT2 PPL ↓
CE62.9115.2512.39
CE + Logit-KD65.7714.4310.31
LoRA65.0814.6811.68
RRD67.5414.059.82

Table R7. Recovery at the main token budget

Qwen2.5-7B E8S2A2Avg. Acc.C4 PPL ↓WT2 PPL ↓
CE55.7518.6316.61
CE + Logit-KD57.6517.4313.76
LoRA54.3920.6017.90
RRD59.7116.8012.99
  • The objective matters beyond the parameterization. With the same setup, RRD is above the strongest baseline (CE + Logit-KD) by 1.77 pp at E8S3A3 and 2.06 pp at E8S2A2, with lower PPL in both.
  • Full fine-tuning alone does not explain the gain. At E8S3A3, full fine-tuning with CE (62.91) is even below CMoE + LoRA (65.08).

At E8S2A2, part of the original gap to CMoE + LoRA (5.32 pp) does come from full fine-tuning and Logit-KD (3.26 pp), and the RRD objectives add the remaining 2.06 pp.

Q2b. "... also add both this control and CMoE + LoRA to the compute-matched Table 3." (Revised: Sec. 4.3, Tab. 3, p. 7, lines 339–368)

While re-checking the FLOPs accounting, we found that the earlier figure (3,370 PFLOPs) was measured on a run without Logit-KD. In the revised Table 3, the cost and the accuracy of RRD come from the same run, which costs about 4,760 PFLOPs. We re-trained every method to this budget.

Table R8. Qwen2.5-7B, 50% FFN sparsity, training budget of about 4,760 PFLOPs

MethodTopologyTrain tokens (Mi)†^\daggerUpdate stepsAvg. Acc.C4 PPL ↓WT2 PPL ↓
CMoE + CEE8S2A2136.3534,90656.2617.2916.45
CMoE + (CE + Logit-KD)E8S2A263.0016,12958.6016.8713.38
CMoE + LoRAE8S2A2228.5658,51154.7019.3018.22
ExpertWeaverE8S2A2120.9430,96154.6819.3018.37
DOT-MoEE8S0A479.2020,27458.3217.7014.60
CMoE + RRDE8S2A240.0010,24059.7116.8012.99

†^\dagger 1 Mi = 2202^{20} tokens

At equal compute, the other methods train on 1.6–5.7× more tokens than RRD. RRD is still the best on all three metrics, 1.11 pp above the next best (CE + Logit-KD).

Reviewer_iQRTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RRD recovers a sparsified, partition-based MoE from a dense teacher while fixing the expert partition and activation budget. It combines teacher-activation-derived routing targets, layerwise FFN-output matching, cross-entropy, and logit distillation. Experiments on Qwen2.5-7B and Llama-2-7B cover two sparsity levels, approximately 40M-token recovery, a compute-matched comparison, downstream teacher-free SFT, and an alternative random partition. RRD consistently improves mean task accuracy over the evaluated pipelines.

Critical strengths
  1. RRD improves mean accuracy across both backbones and sparsity levels (Tables 1–2). On Qwen2.5-7B at 50% sparsity, it also exceeds DOT-MoE by 2.47 points under the reported approximate training-FLOP budget (Table 3). This supports its usefulness beyond a single favorable configuration.

  2. Gains largely persist through identical teacher-free SFT, and RRD substantially improves recovery from a random expert partition without retuning (Tables 4–5). These experiments strengthen the case that the recovery procedure is useful beyond its primary CMoE initialization.

Critical weaknesses
  1. Figure 2 never removes routing distillation or residual matching individually. Moreover, CE/logit losses cannot update the router under the stated unweighted hard selection (§3.4). Their comparison with RRD therefore combines adding a router-learning signal with adding residual supervision. The results do not establish whether residual matching contributes beyond supervised routing, or whether both proposed objectives are needed. This is consequential to the paper's central methodological contribution.

  2. RRD updates full expert weights, whereas the strongest same-topology CMoE baseline uses rank-8 LoRA (Appendices B–C). Other baselines additionally change conversion procedures. The controlled objective comparisons use only 4M tokens, rather than the main 40M-token budget. Consequently, the main gains cannot be cleanly attributed to RRD's objectives rather than the recovery parameterization and training recipe.

Questions for authors

Does the approximately 3,370-PFLOP accounting in Table 3 include the second, teacher-forced student forward pass and router backward computation in Algorithm 1? Please provide the teacher/student/router cost breakdown; it would determine how strongly the compute-matched result supports the efficiency claim.

5 comments

Thank you for your interest in our work and for the thoughtful review.

The revised version addresses your comments with the following additions.

  • We re-ran the Fig. 2 ablation study with 40M C4 tokens, the same budget as the main experiments.
  • We added variants that add routing distillation (RouDistill) and residual distillation (ResDistill) to CE + Logit-KD individually (W1).
  • We added an experiment that tests whether the output-loss gradients need to be detached from the router (W1).
  • After CMoE conversion, we added and compared recovery with CE and with CE + Logit-KD, in addition to LoRA (W2).
  • We clarified the compute-matched experiment and added CMoE + LoRA, CE, and CE + Logit-KD to it (Q1).

All results below use Qwen2.5-7B E8S2A2 (50% FFN sparsity) and 40M C4 tokens.

W1. "Figure 2 never removes routing distillation or residual matching individually. Moreover, CE/logit losses cannot update the router under the stated unweighted hard selection ..."

This is a fair point, and we agree that it needs to be verified. We ran two additional experiments. The first evaluates routing distillation (RouDistill) and residual distillation (ResDistill) separately (W1a). The second trains the router without detaching the gradients of the CE, Logit-KD, and ResDistill objectives (W1b).

W1a. Individual effects of RouDistill and ResDistill
(Revised manuscript: Sec. 4.6, Fig. 2-3, p. 8-9, lines 421-468)

Table R1. Adding each objective to CE + Logit-KD (unweighted top-K, as in the paper)

Qwen2.5-7B E8S2A2Avg. Acc.C4 PPL ↓WT2 PPL ↓
CE + Logit-KD57.4917.9214.28
+ RouDistill59.4716.9713.32
+ ResDistill57.4417.7113.96
+ RouDistill + ResDistill (RRD)59.7116.8012.99
  • RouDistill gives most of the accuracy gain (+1.98 pp of the total +2.22 pp).
  • ResDistill on top of RouDistill improves all three metrics (C4 PPL 16.97 → 16.80, WT2 PPL 13.32 → 12.99, accuracy +0.24 pp). The accuracy gain is small, and the PPL gain is the clearer effect.
  • ResDistill alone improves PPL but not accuracy. In this row the router is frozen at its initialization. Once the router is trained, ResDistill also improves accuracy (+1.27 pp, W1b).

W1b. Controls where the router is trained by the output losses
(Revised manuscript: Sec. 4.7, Tab. 6, p. 9, lines 471-485; Appendix D.3, Tab. 12-13, p. 19-20, lines 1019-1057)

We also evaluated the suggested alternative, a differentiable weighted top-K forward through which the expert objectives can train the router.

Table R2. Controls with a trainable router

ObjectivesAggregationRouter gradientsAvg. Acc.C4 PPL ↓WT2 PPL ↓
CEUnweightedFrozen54.7819.0017.30
CEWeightedCE55.7518.6316.61
CE + Logit-KDUnweightedFrozen57.4917.9214.28
CE + Logit-KDWeightedCE + Logit-KD57.6517.4313.76
CE + Logit-KD + ResDistillUnweightedFrozen57.4417.7113.96
CE + Logit-KD + ResDistillWeightedCE + Logit-KD + ResDistill58.9217.2413.30
CE + Logit-KD + RouDistillWeightedAll enabled losses58.4417.5713.86
CE + Logit-KD + RouDistillUnweightedRouDistill only$^\ast$59.4716.9713.32
RRDWeightedAll losses59.0117.4113.40
RRDUnweightedRouDistill only$^\ast$59.7116.8012.99

∗^\ast The setting used in the paper. "Unweighted" is a plain top-K sum, and "Weighted" is a softmax-weighted top-K sum.

  • Output losses do train the router, but less effectively than RouDistill. A router trained by the output losses beats a frozen one in every pair, yet the best such variant (58.92) is still below RRD (59.71).
  • Adding the output-loss gradients to the router does not help. With RouDistill present, the weighted variant that also sends the output losses to the router is lower (59.47 → 58.44, 59.71 → 59.01).

In summary, both objectives contribute, and RouDistill contributes more than ResDistill. The current design, which uses both and trains the router with RouDistill only, gives the best result.

W2. "... the main gains cannot be cleanly attributed to RRD's objectives rather than the recovery parameterization and training recipe."
(Revised manuscript: Sec. 4.7, Tab. 6, p. 9, lines 470-485; Appendix C, p. 17, lines 903-907; Appendix D.3, Tab. 12, p. 20, lines 1030-1037)

The ablation study in Sec. 4.6 was meant to show the effect of RRD under the same CMoE conversion, but we acknowledge that it used 4M C4 tokens, which differs from the main budget. As noted above, we re-ran the ablation study with 40M training tokens.

We also added variants that, after CMoE conversion, train with CE or with CE + Logit-KD and update the router through weighted aggregation (Table R2).

Table R3. Same conversion and training setup, different objectives

Qwen2.5-7B E8S2A2TrainableAvg. Acc.C4 PPL ↓WT2 PPL ↓
CMoE + LoRA (reference)Rank-8 LoRA54.3920.6017.90
CMoE + CEFull experts + router55.7518.6316.61
CMoE + (CE + Logit-KD)Full experts + router57.6517.4313.76
CMoE + RRDFull experts + router59.7116.8012.99

Table R3 agrees with Table R2. The 5.32 pp gap between CMoE + LoRA and RRD breaks down as follows.

  • +1.36 pp from full fine-tuning of the experts and the router instead of LoRA (54.39 → 55.75)
  • +1.90 pp from Logit-KD (55.75 → 57.65)
  • +2.06 pp from the two RRD objectives (57.65 → 59.71), with lower PPL on both datasets

Q1a. "Does the approximately 3,370-PFLOP accounting in Table 3 include the second, teacher-forced student forward pass and router backward computation in Algorithm 1?"
(Revised manuscript: Sec. 4.3, Tab. 3, p. 7, lines 339-368)

Yes, both were included. The 3,370 PFLOPs figure, however, was measured on a run without Logit-KD, so it did not correspond to the run that produced the reported accuracy. In the revised Table 3, the cost and the accuracy of RRD come from the same run. Its cost is about 4,760 PFLOPs, and we re-trained every baseline to this budget.

The increase of about 1,391 PFLOPs has two parts. Logit-KD accounts for 1,026 (its backward pass and the teacher's LM head). The remaining 365 is implementation overhead that RRD does not need. The current code recomputes the teacher FFN activations for the routing target (319) and runs the LM head in the second student forward (46).

Table R4. Qwen2.5-7B, 50% FFN sparsity, training budget of about 4,760 PFLOPs

MethodTopologyTrain tokens (Mi)†^\daggerUpdate stepsAvg. Acc.C4 PPL ↓WT2 PPL ↓
CMoE + CEE8S2A2136.3534,90656.2617.2916.45
CMoE + (CE + Logit-KD)E8S2A263.0016,12958.6016.8713.38
CMoE + LoRAE8S2A2228.5658,51154.7019.3018.22
ExpertWeaverE8S2A2120.9430,96154.6819.3018.37
DOT-MoEE8S0A479.2020,27458.3217.7014.60
CMoE + RRDE8S2A240.0010,24059.7116.8012.99

†^\dagger 1 Mi = 2202^{20} tokens ("40M" in the paper means 40 Mi).

At equal compute, the other methods train on 1.6-5.7× more tokens than RRD. RRD is still the best on all three metrics, 1.11 pp above the next best (CE + Logit-KD).

Q1b. "Please provide the teacher/student/router cost breakdown"
(Revised manuscript: Appendix B.1, Tab. 7, p. 15-16, lines 802-825)

Table R5. RRD training cost breakdown (40 Mi tokens)

ComponentPFLOPs
Teacher forward and routing-target construction946.89
Ordinary student forward and loss construction397.40
Expert backward for CE, including recomputation980.20
Expert backward for Logit-KD, including recomputation980.31
Expert backward for ResDistill, including recomputation934.54
Teacher-forced student forward397.19
Router backward, including recomputation120.81
Gradient clipping and optimizer1.71
Total4,759.04
  • Routing distillation costs 10.9% in total, most of it the second, teacher-forced forward (8.3%).
  • The student backward dominates (60.8%). Our implementation back-propagates CE, Logit-KD, and ResDistill separately, each with its own checkpointing recomputation. A single backward on the weighted sum gives the same gradients at lower cost, so 4,760 PFLOPs is an upper bound on RRD's cost and the comparison in Table R4 is conservative for RRD.

We are writing to kindly follow up on our response to your review.
We know this is a busy period, and we are grateful for the time you have given to our paper.

In our response, we did our best to address each of your concerns with quantitative results and qualitative analysis, and all of these additions are reflected in the revised manuscript.

If any of your concerns are still unresolved, we would be grateful if you could let us know, and we will do our best to address them before the discussion period ends.
If you feel they have been addressed, we would appreciate it if you would kindly consider reflecting this in your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all reviewers (iQRT, n8Rr, gsHh) for their constructive and thorough feedback.

Strengths Highlighted by Reviewers

  • Consistent gains (iQRT, n8Rr): Improves accuracy across Qwen2.5-7B/Llama-2-7B, both sparsities, and FLOP-matched runs.
  • Transferability (iQRT, n8Rr, gsHh): Gains persist after teacher-free SFT and hold on random-split LLaMA-MoE.
  • Clarity (gsHh): Clear training recipe with complementary SFT, random-split, and proximity analyses.

Key Takeaways

  • RRD's core contribution is combining RouDistill and ResDistill with separated gradient paths (only RouDistill updates the router; output losses update experts).
  • Controlled gains over full-FT CE+Logit-KD grow with sparsity (+1.77 → +2.06 → +3.09 pp) and hold at matched FLOPs.

Unless noted, controlled runs use Qwen2.5-7B E8S2A2 (50% sparsity, 40M C4 tokens).

1. Core Ablations: Individual Losses, 40M Budget & Router Gradients (iQRT, n8Rr, gsHh)

  • Individual Losses at 40M (iQRT W1, n8Rr W4/Q4, gsHh W5/W6):
    • Fig. 2 previously tested both losses only jointly at 4M tokens; we re-ran Fig. 2-3 at 40M with single-loss variants.
    • Over trainable-router CE+Logit-KD (57.65%), RouDistill adds +1.82 pp and ResDistill adds +1.27 pp; RRD (both) is best on all metrics (59.71%). (Sec. 4.6, Fig. 2-3; App. D.3, Tab. 13)
  • Trainable-Router Controls & Separated Gradients (iQRT W1, n8Rr W2/W4/Q1):
    • Unweighted top-K leaves non-RouDistill routers frozen, so we added softmax-weighted controls; output-loss router training beats frozen routers yet trails RRD (58.92% vs. 59.71%) and keeps routing hit rate at chance (0.334 vs. 0.638).
    • Sending output-loss gradients to the router with RouDistill hurts (59.71% → 59.01%), confirming separated gradient paths are essential. (Sec. 4.7, Tab. 6; App. D.3, Tab. 12-13)

2. Controlled Baselines & Compute-Matched Accounting (iQRT, n8Rr)

  • Full-FT vs. LoRA (iQRT W2, n8Rr W3/Q2):
    • To disentangle full expert FT from rank-8 LoRA, we added identical full-FT CE and CE+Logit-KD controls.
    • At E8S2A2, the 5.32 pp gap over LoRA splits into +1.36 pp (full FT), +1.90 pp (Logit-KD), and +2.06 pp (RRD losses) with lower PPL; at E8S3A3, RRD leads by +1.77 pp. (Sec. 4.7, Tab. 6; App. C; App. D.3, Tab. 12; E8S3A3: n8Rr Q2a)
  • Re-Audited FLOPs & Breakdown (iQRT Q1, n8Rr W3/Q2):
    • 3,370 PFLOPs omitted Logit-KD; we re-trained all baselines (incl. LoRA/full-FT) to the exact 40M RRD cost (4,760 PFLOPs, incl. teacher, TF pass, router backward).
    • Despite 1.6-5.7× more tokens, RRD still leads (59.71% vs. 58.60%, +1.11 pp). RouDistill is 10.9%, and separate expert backwards (60.8%) make 4,760 PFLOPs conservative. (Sec. 4.3, Tab. 3; App. B.1, Tab. 7)

3. Prior Works & Design Choice Ablations (n8Rr)

  • MoEfication & LaDiMo (n8Rr W1):
    • Unlike MoEfication (frozen router) and LaDiMo (local outputs), RRD trains both with separated gradients.
    • RRD beats MoEfication on E592S0A296 in accuracy (+0.71 pp, 65.30%) and WT2 PPL (MoEfication leads C4), and beats LaDiMo-style KD on Llama-2-7B by +1.41 pp (66.47%) with lower PPL. (App. D.2, Tab. 10-11)
  • Design Choices, S∗S^\ast & Router (n8Rr W2/Q3):
    • TF vs. ordinary pass: TF is an implementation choice, not a claimed contribution (8.3% cost in Tab. 3); final accuracy is comparable (59.71% vs. 59.88%). (n8Rr Q3a)
    • Own-path vs. shared-input: Shared-input matches C4, while RRD (own-path) achieves statistically significantly lower WT2 PPL. (Sec. 4.9, Fig. 5; App. D.3, Tab. 16, Fig. 7)
    • Oracle routing (S∗S^\ast): Test-time S∗S^\ast lifts RRD by +3.04 pp (62.75%), validating summed absolute activation as a routing target. (App. D.1, Tab. 9)
    • Router: 2-layer MLP (d → 1,024 → GELU → routed), explaining the 0.07-0.11B param gap. (App. B; App. C, Tab. 8)

4. High Sparsity & Fixed-Teacher Dynamics (gsHh)

  • 75% Sparsity Fine-Grained MoE (gsHh W1/W4):
    • On E32S1A7 (75% sparsity, 32 experts, 1 shared + 7 routed), RRD reaches 50.76%, beating CE+Logit-KD (+3.09 pp) and LoRA (+7.07 pp) with lowest PPL. (App. D.4, Tab. 17-18)
  • Fixed-Teacher & Refresh (gsHh W2/W3):
    • Midpoint (20M) switching shows removing distillation late hurts (-0.99 to -2.52 pp) while adding it helps (+1.07 to +1.78 pp).
    • Reassembled dense models have worse PPL than the original teacher for all methods, so refreshing the teacher degrades target quality. (Sec. 4.8, Fig. 4; App. D.3, Tab. 14-15, Fig. 6)

We hope this summary clarifies how all reviewer comments have been addressed in the revised manuscript and individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.