- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates independent post-training and reintegration of domain experts in pretrained mixture-of-experts models. The proposed approach extracts domain-specific expert subsets using calibration data, updates the selected routed experts while freezing the entire interface, and reinserts the updated experts into their original slots. Independently trained modules are composed by writing their experts into a common base model, with overlapping slots resolved through weight averaging or domain priority. Experiments primarily on EMO examine reinsertion loss, training resource requirements, and the retention of domain-specific gains after composition.
Critical strengths
- The paper addresses a relevant practical problem and provides evidence that expert–interface co-adaptation contributes to reinsertion loss. The comparison against training the same experts within the full model is particularly informative. Extracted training reduces resident parameters from 14B to 3.9B while incurring an accuracy reduction of 0–1.5 percentage points on the reported tasks. These results support the practical value of the approach for pretrained MoEs with suitable modular structure.
Critical weaknesses
-
The evidence does not establish the necessity of freezing the entire interface. Restoring fine-tuned interface components after training and selectively freezing those components during training are different interventions. The restoration experiments therefore do not determine which components must remain fixed or whether freezing the entire interface sacrifices useful adaptation unnecessarily.
-
The absence of overlapping expert slots eliminates direct parameter overwrite conflicts, but it does not rule out functional interference through routing, hidden representations, or interactions across layers. The association between subset overlap and gain retention is also confounded by domain identity, as the authors acknowledge for low-overlap pairs involving history.
Questions for authors
-
Could the authors compare full-interface freezing with selective freezing during training, including freezing only attention or only the routers and shared experts? Reporting both standalone and reinserted accuracy under the same expert-only reinsertion protocol would clarify whether freezing the entire interface is necessary and whether a better adaptation–reinsertion trade-off exists.
-
Could the authors provide controlled evidence distinguishing interference caused by overlapping slots from other functional interactions? In particular, the statement that interference can arise only in shared slots requires stronger justification or narrower wording.
-
The frequent use of semicolons to connect multiple claims, together with repeated qualifications within the same paragraphs, makes the argument difficult to follow.
6 comments
Q2: Functional interference and the history confound
We agree on both points:
-
Narrower claim. The revised §3.4 reads: "Parameter conflicts arise only in shared slots, although updates in disjoint slots can still interact through the shared computation." The sentence is revised in all three places where it appeared (§2, §3.4, §4.3.4).
-
Controlled test of functional interference. We hold module a's parameters fixed and keep the shared slots at their original experts. Adding the partner's non-shared experts then changes accuracy on domain a by at most 0.7 points, and no interval excludes zero. In high-overlap pairs, averaging the shared slots costs 1.8–2.6 points. Composition loss in our tests is therefore concentrated in the shared slots. The test does not separate routing, hidden-representation, and cross-layer pathways.
-
History confound. We varied overlap with the partner domain fixed. With history as math's partner, raising the shared slots from 99 to 347 lowers math's composed accuracy by 2.1 [−4.2, 0.0] points, while history's own gain stays similar. Pairs with law instead of history also retain more than high-overlap pairs, within the CIs. The same control shows that the slot count is not the whole story: which slots are shared also matters. The revision therefore states the overlap–retention relation as an association, not a cause.
Q3: Writing
We agree, and we thank the reviewer for the specific diagnosis. The original main text contained 43 semicolons, 20 of them in the experiments section. The revision makes two changes:
- One claim per sentence. Sentences that joined claims with semicolons are split. This removes 42 of the 43 semicolons, and the list of baselines in §4.1 becomes an itemized list.
- Qualifications in one place. Where a paragraph interleaved results with repeated qualifications, the result now comes first, and the qualifications are collected into one scope statement. The composition paragraph of §4.3.4 is the main example.
An example from §3.3:
Before. "We do not claim novelty for freezing the parameters outside the routed experts as such; our contribution is to show why the freeze is necessary for reinsertion and that it is sufficient, i.e., that training in the extracted module loses almost nothing relative to training inside the full model."
After. "We do not claim novelty for freezing the parameters outside the routed experts as such. Our contribution is to show why reinsertion requires freezing the interface components that the experts co-adapt with, and that training in the extracted module then loses almost nothing relative to training inside the full model. Freezing attention, the routers, and the shared expert together works as well as freezing the whole interface, and freezing the normalization layers, embeddings, and output head makes no measurable difference (Appendix L). We freeze the whole interface as the simplest rule."
We hope the revised version is easier to follow, and we would welcome any passage the reviewer still finds unclear.
We thank Reviewer
xSUxfor the careful review and concrete suggestions! We especially appreciate their recognition that the paper "addresses a relevant practical problem", "provides evidence that expert-interface co-adaptation contributes to reinsertion loss," and that "the comparison against training the same experts within the full model is particularly informative." We also value their assessment that the results "support the practical value of the approach for pretrained MoEs with suitable modular structure."Below, we answer each question. We ran the selective-freezing experiment proposed in Q1 and three composition experiments for Q2. We would be glad to discuss any further questions!
Q1: Is freezing the entire interface necessary?
We agree that restoring interface components after training and freezing them during training are different interventions. We therefore ran the comparison you proposed.
Routed experts are always trained. All variants are reinserted with the same expert-only protocol. GSM8K uses the short schedule with three seeds, averaged per question. Math and physics use the main recipe with one seed. Gap = reinserted − standalone module, with paired 95% CIs. All variants, including IF-FT and FP-FT, were trained and evaluated together, so absolute accuracies differ slightly from Table 1, and we compare only within this table.
On physics, the FP-FT gap itself is not significant, and all variants are within noise.
Freezing the entire interface performs the same, within noise, as the smallest sufficient freeze we found. None of the tested partial freezes gives a better trade-off between adaptation and reinsertion. We keep the full freeze as the simplest rule. The revision replaces "necessary and sufficient" in §3.3 with this result and adds Appendix L, which also lists standalone and reinserted accuracy for math and physics.