acceptodds
Under review as a conference paper at ICLR 2027

MoE Modular Post-Training: Improving and Composing Emergent Domain Experts

Abstract

Modular pretraining enables Mixture-of-Experts (MoE) language models to form domain expert modules that can operate independently. Can these modules also be independently improved and composed? We study MoE modular post-training: training domain modules separately and integrating their updates into a shared model. We find that modules fine-tuned with all their parameters can lose much of their improvement when reinserted. Controlled interventions trace this loss not to router mismatch but primarily to co-adaptation between the domain experts and the model's interface, the parameters outside the routed experts: both change during fine-tuning, but only the expert updates are reinserted. We therefore introduce Interface-Frozen Fine-Tuning (IF-FT), which updates only the domain experts and keeps the interface fixed. Across seven domains on EMO, a 14B-parameter modular MoE, IF-FT preserves standalone accuracy after reinsertion to within test-set noise and improves reinserted accuracy by up to 10 points over full-parameter module fine-tuning. It closely matches training the same experts inside the full model while holding only 3.9B of the 14B parameters in memory during training, and it does not reduce accuracy on other domains. Independently trained modules also compose without further training, and no domain falls below the untrained model; their gains survive best when the modules share few experts. In a standard MoE trained on the same data, reinsertion is equally lossless, but even after training the extracted module falls below its own full model. Emergent domain experts can thus serve as independent units of post-training and composition, provided modularity is built in during pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Expert, modular MoE, post-training, expert-specialized fine-tuning, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xSUxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates independent post-training and reintegration of domain experts in pretrained mixture-of-experts models. The proposed approach extracts domain-specific expert subsets using calibration data, updates the selected routed experts while freezing the entire interface, and reinserts the updated experts into their original slots. Independently trained modules are composed by writing their experts into a common base model, with overlapping slots resolved through weight averaging or domain priority. Experiments primarily on EMO examine reinsertion loss, training resource requirements, and the retention of domain-specific gains after composition.

Critical strengths
  1. The paper addresses a relevant practical problem and provides evidence that expert–interface co-adaptation contributes to reinsertion loss. The comparison against training the same experts within the full model is particularly informative. Extracted training reduces resident parameters from 14B to 3.9B while incurring an accuracy reduction of 0–1.5 percentage points on the reported tasks. These results support the practical value of the approach for pretrained MoEs with suitable modular structure.
Critical weaknesses
  1. The evidence does not establish the necessity of freezing the entire interface. Restoring fine-tuned interface components after training and selectively freezing those components during training are different interventions. The restoration experiments therefore do not determine which components must remain fixed or whether freezing the entire interface sacrifices useful adaptation unnecessarily.

  2. The absence of overlapping expert slots eliminates direct parameter overwrite conflicts, but it does not rule out functional interference through routing, hidden representations, or interactions across layers. The association between subset overlap and gain retention is also confounded by domain identity, as the authors acknowledge for low-overlap pairs involving history.

Questions for authors
  1. Could the authors compare full-interface freezing with selective freezing during training, including freezing only attention or only the routers and shared experts? Reporting both standalone and reinserted accuracy under the same expert-only reinsertion protocol would clarify whether freezing the entire interface is necessary and whether a better adaptation–reinsertion trade-off exists.

  2. Could the authors provide controlled evidence distinguishing interference caused by overlapping slots from other functional interactions? In particular, the statement that interference can arise only in shared slots requires stronger justification or narrower wording.

  3. The frequent use of semicolons to connect multiple claims, together with repeated qualifications within the same paragraphs, makes the argument difficult to follow.

6 comments

We thank Reviewer xSUx for the careful review and concrete suggestions! We especially appreciate their recognition that the paper "addresses a relevant practical problem", "provides evidence that expert-interface co-adaptation contributes to reinsertion loss," and that "the comparison against training the same experts within the full model is particularly informative." We also value their assessment that the results "support the practical value of the approach for pretrained MoEs with suitable modular structure."

Below, we answer each question. We ran the selective-freezing experiment proposed in Q1 and three composition experiments for Q2. We would be glad to discuss any further questions!

Q1: Is freezing the entire interface necessary?

We agree that restoring interface components after training and freezing them during training are different interventions. We therefore ran the comparison you proposed.

Routed experts are always trained. All variants are reinserted with the same expert-only protocol. GSM8K uses the short schedule with three seeds, averaged per question. Math and physics use the main recipe with one seed. Gap = reinserted − standalone module, with paired 95% CIs. All variants, including IF-FT and FP-FT, were trained and evaluated together, so absolute accuracies differ slightly from Table 1, and we compare only within this table.

Frozen during trainingGSM8K standaloneGSM8K reinsertedGSM8K gapMath gapPhysics gap
Nothing (FP-FT)21.412.3−9.1 [−11.1, −7.1]−4.4 [−8.2, −0.7]−3.1 [−7.4, +1.2]
Norms, embeddings, output head21.011.9−9.1 [−11.1, −7.1]−4.0 [−7.7, −0.2]−1.6 [−5.8, +2.7]
Attention19.013.4−5.6 [−7.5, −3.7]−4.0 [−7.0, −0.9]−1.6 [−4.7, +1.6]
Routers + shared expert19.112.8−6.3 [−8.2, −4.4]−1.9 [−5.2, +1.4]−1.2 [−5.1, +2.7]
Attention + routers + shared expert16.516.1−0.4 [−2.2, +1.5]−0.9 [−3.3, +1.4]−0.8 [−3.9, +2.3]
Entire interface (IF-FT)16.915.9−1.0 [−2.9, +0.9]+1.4 [−1.2, +4.0]−0.4 [−3.5, +2.7]
  1. Attention and the routers with the shared expert must be frozen together. Freezing either group alone leaves 62–70% of the FP-FT gap on GSM8K. The reinserted model is then significantly below IF-FT: −2.5 [−4.1, −1.0] when only attention is frozen, and −3.1 [−4.8, −1.5] when only the routers and shared expert are frozen. On math, freezing only attention also leaves a significant gap.
  2. Normalization layers, embeddings, and output head make no measurable difference. Freezing only these behaves like FP-FT. Training only these, with the rest of the interface frozen, behaves like IF-FT: its reinserted GSM8K accuracy differs from IF-FT by +0.2 [−0.5, +0.9].
  3. Partial freezing gives stronger standalone modules, not better reinserted models. On GSM8K, freezing only attention, or only the routers and shared expert, raises standalone accuracy by about 2 points over IF-FT. This adaptation is lost at reinsertion. Under expert-only reinsertion, every interface update is discarded at deployment, and experts that adapted to it perform worse.

On physics, the FP-FT gap itself is not significant, and all variants are within noise.

Conclusion.

Freezing the entire interface performs the same, within noise, as the smallest sufficient freeze we found. None of the tested partial freezes gives a better trade-off between adaptation and reinsertion. We keep the full freeze as the simplest rule. The revision replaces "necessary and sufficient" in §3.3 with this result and adds Appendix L, which also lists standalone and reinserted accuracy for math and physics.

Q2: Functional interference and the history confound

We agree on both points:

  • Narrower claim. The revised §3.4 reads: "Parameter conflicts arise only in shared slots, although updates in disjoint slots can still interact through the shared computation." The sentence is revised in all three places where it appeared (§2, §3.4, §4.3.4).

  • Controlled test of functional interference. We hold module a's parameters fixed and keep the shared slots at their original experts. Adding the partner's non-shared experts then changes accuracy on domain a by at most 0.7 points, and no interval excludes zero. In high-overlap pairs, averaging the shared slots costs 1.8–2.6 points. Composition loss in our tests is therefore concentrated in the shared slots. The test does not separate routing, hidden-representation, and cross-layer pathways.

  • History confound. We varied overlap with the partner domain fixed. With history as math's partner, raising the shared slots from 99 to 347 lowers math's composed accuracy by 2.1 [−4.2, 0.0] points, while history's own gain stays similar. Pairs with law instead of history also retain more than high-overlap pairs, within the CIs. The same control shows that the slot count is not the whole story: which slots are shared also matters. The revision therefore states the overlap–retention relation as an association, not a cause.

Q3: Writing

We agree, and we thank the reviewer for the specific diagnosis. The original main text contained 43 semicolons, 20 of them in the experiments section. The revision makes two changes:

  1. One claim per sentence. Sentences that joined claims with semicolons are split. This removes 42 of the 43 semicolons, and the list of baselines in §4.1 becomes an itemized list.
  2. Qualifications in one place. Where a paragraph interleaved results with repeated qualifications, the result now comes first, and the qualifications are collected into one scope statement. The composition paragraph of §4.3.4 is the main example.

An example from §3.3:

Before. "We do not claim novelty for freezing the parameters outside the routed experts as such; our contribution is to show why the freeze is necessary for reinsertion and that it is sufficient, i.e., that training in the extracted module loses almost nothing relative to training inside the full model."

After. "We do not claim novelty for freezing the parameters outside the routed experts as such. Our contribution is to show why reinsertion requires freezing the interface components that the experts co-adapt with, and that training in the extracted module then loses almost nothing relative to training inside the full model. Freezing attention, the routers, and the shared expert together works as well as freezing the whole interface, and freezing the normalization layers, embeddings, and output head makes no measurable difference (Appendix L). We freeze the whole interface as the simplest rule."

We hope the revised version is easier to follow, and we would welcome any passage the reviewer still finds unclear.

Reviewer_W8iCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether domain-specific expert subsets of a modular Mixture-of-Experts model can be fine-tuned independently, reinserted into the full model, and composed across domains. On EMO, full-parameter fine-tuning of an extracted module loses some of its gains when only the updated experts are reinserted. Interventions that restore or freeze interface components support expert–interface co-adaptation as a major explanation. The proposed Interface-Frozen Fine-Tuning (IF-FT) updates only the selected routed experts. Experiments cover six MMLU domains and GSM8K, module composition, and standard-MoE comparisons. On the evaluated EMO settings, extracted-module training approaches in-place training while reducing resident parameters, and independently trained modules preserve varying fractions of their individual gains after composition.

Critical strengths

The paper identifies and investigates a meaningful module-training problem. Whether improvements learned by an extracted module survive reinsertion is central to modular post-training. Freezing the interface and restoring the fine-tuned interface provide complementary interventions supporting the co-adaptation explanation. This diagnosis goes beyond reporting an accuracy gap and helps explain the mismatch between independent training and deployment within the original model.

Independent training and reinsertion have practical potential. On the evaluated EMO settings, IF-FT reduces resident parameters from 14B to 3.9B while approaching the accuracy of training the same experts inside the full model. Composing independently trained modules without further optimization offers a plausible path toward separately maintained domain capabilities. The capacity-matched ESFT experiments and negative results on standard MoEs help identify the method’s contribution and its dependence on pretrained modularity.

Critical weaknesses

The main comparison should distinguish accuracy gains from the residency benefit of extraction.

The default ESFT baselines in Table 2 (p. 7) train far fewer experts than IF-FT. Appendix F and Table 10 (pp. 14–15) provide an important control: selecting 31 experts per layer with ESFT reaches accuracy comparable to in-place IF-FT. Section 3.3 (p. 5) also acknowledges that freezing non-routed-expert parameters is not itself novel. These comparisons limit attribution of the main-table accuracy advantage to the fine-tuning rule, while leaving the reinsertion diagnosis and reduced residency through extraction as substantive contributions. Appendix E (p. 14) reports LoRA underfitting under the shared recipe, while Appendix G (p. 15) identifies implementation dependence in the speedup. Both limitations are disclosed, but should be reflected more prominently in the main comparison. The resident-parameter saving is established; the practical accuracy–cost advantage over comparably configured alternatives is not yet fully characterized.

The scope of losslessness and the interpretation of composition need more precise support.

Section 3.1 (p. 4) defines losslessness using a one-standard-error threshold and summarizes other-domain changes by their mean. This threshold does not establish paired non-inferiority, and a positive average does not rule out individual regressions. Appendix F (p. 14) reports a small MMLU-math decrease after GSM8K training, although its significance is unclear. Section 3.4 (p. 5) further states that interference can arise only in shared slots. Disjoint updates avoid direct parameter overwriting, but do not establish functional independence through the shared computation. Whether such interactions contribute to composition loss remains untested. The overlap analysis also has an acknowledged confound: all three lowest-overlap pairs contain history (pp. 7–9). Consequently, the observed composition results do not yet establish overlap as an independent predictor. Paired uncertainty estimates and a narrower statement of the interference claim would materially strengthen the conclusions.

Questions for authors

Contribution attribution: Can the existing capacity-matched ESFT results in Table 10 be brought into the main comparison, explicitly separating expert-selection accuracy from the residency benefit of extraction? Please clarify which accuracy and efficiency advantages remain under this comparison.

Statistical meaning of losslessness: Can the authors specify a practically meaningful non-inferiority margin and report paired confidence intervals for module–reinsertion differences using per-question predictions and the available reruns? Which domains support non-inferiority, and which remain inconclusive?

Composition mechanism: Does the shared-slot claim refer only to parameter overwriting? Is there evidence separating overlap effects from domain identity, such as a fixed-pair overlap control or low-overlap pairs without history? If not, can the mechanism claim be restricted to the observed association?

Non-target performance and applicability: Can per-domain changes and their uncertainty be reported alongside averages? Please also distinguish non-inferiority relative to the trained module from improvement over the original full model, particularly when discussing the standard-MoE results in Appendix I.

Practical efficiency: Can the authors provide measured peak memory and end-to-end training time, and clarify the tuning budget for LoRA? If optimized-kernel results are unavailable, please keep the 3.5× speedup explicitly implementation-specific. These details would affect the practical significance of the established residency saving.

5 comments

We thank Reviewer W8iC for the detailed and constructive review! We especially appreciate their recognition that the paper "identifies and investigates a meaningful module-training problem," that freezing and restoring the interface "provide complementary interventions supporting the co-adaptation explanation," and that "this diagnosis goes beyond reporting an accuracy gap." We also value their view that "independent training and reinsertion have practical potential" and that the capacity-matched ESFT and standard-MoE results "help identify the method's contribution and its dependence on pretrained modularity."

The questions were precise. We answer each with paired analyses and new experiments, and we would be glad to discuss further. All intervals are paired 95% bootstrap CIs over test questions, in accuracy points. "Pooled" averages the five domains (math, physics, health, history, GSM8K) with equal weight.

Q1: Contribution attribution

We agree that the main comparison should separate the two. Table 2 already contains the strictest capacity-matched control: the same experts trained inside the full model ("IF-FT without extraction"). ESFT capped at 31 experts per layer (Table 10) selects 84–91% of the same experts, so its rows nearly duplicate that control (+0.6 [−0.0, +1.2]). Instead of adding near-duplicate rows, the revised §4.3.3 states the attribution: the advantage over default ESFT comes from the number of experts the thresholds admit (17–43 vs 496), not from ESFT's scoring rule. It also states that at matched capacity IF-FT is not more accurate than ESFT trained inside the full model, and that its advantage is the smaller memory footprint and a standalone module.

We also ran the missing cell: ESFT-Gate's experts capped at 31 per layer, trained with IF-FT in an extracted module and reinserted. This completes a 2×2 design. Mean reinserted accuracy over the five domains (one seed):

Expert subsetExtracted module (IF-FT)Inside the full model
Ours (average routing probability)42.143.0
ESFT-Gate, 31 per layer41.543.6

Paired effects, pooled:

  • Cost of extraction, our subsets: −0.9 [−1.9, +0.1], with no single domain significant. It is largest on GSM8K (about −2).
  • Cost of extraction, ESFT's subsets: −2.0 [−3.1, −1.0]. The difference from ours is −1.2 [−2.1, −0.2].
  • ESFT's selection vs ours, inside the full model: +0.6 [−0.0, +1.2].
  • ESFT's selection vs ours, extracted: −0.6 [−1.3, +0.1].
  • Default-threshold ESFT vs IF-FT: −4.6 [−5.8, −3.3] (ESFT-Gate) and −3.1 [−4.3, −2.0] (ESFT-Token).

The new cell was trained on a different GPU type. Evaluating the untrained model on both GPU types gives identical predictions on every test question. The extracted cell with our subsets is a rerun on the same GPU type as the full-model rows and differs from Table 2 by at most 0.5 points.

What remains under this comparison.

  • Accuracy. At matched capacity, IF-FT has no accuracy advantage. ESFT's selection helps slightly inside the full model, but not in an extracted module.
  • Memory. IF-FT holds 3.9B instead of 13.6B parameters. Measured peak memory is 32.8 vs 52.1 GB (Q5).
  • A standalone module. The module can be trained and served without the full model.
  • Diagnosis. We explain why reinsertion loses the gain and which interface components must stay frozen (R3 Q1). ESFT never reinserts, so it does not face this problem.

Composition by slot requires only that training updates routed experts alone, so it also applies to experts trained inside the full model.

Q2: Statistical meaning of losslessness

We now test paired non-inferiority. The margin is δ = 2 points per domain. This is about one unpaired standard error of an MMLU domain test set, and less than half of the single-module gains in the reasoning-heavy domains and health (5–13 points). For pooled tests we use the stricter δ = 1. Reinsertion is non-inferior if the lower CI bound exceeds −δ.

Reinsertion gap (reinserted − standalone module), main runs:

DomainIF-FTFP-FT
GSM8K−1.1 [−3.5, +1.4]−16.7 [−19.4, −14.0]
Physics0.0 [−3.1, +3.1]−3.9 [−8.2, +0.4]
Math+1.2 [−1.2, +3.5]−2.8 [−6.6, +0.9]
Health+1.4 [−0.9, +3.6]0.0 [−2.7, +2.7]
History+0.8 [−2.1, +3.8]+1.6 [−2.1, +5.4]
Psychology+0.6 [−2.2, +3.5]+2.6 [−0.4, +5.6]
Law−0.6 [−2.7, +1.6]−0.4 [−2.8, +2.1]
Pooled, all seven+0.3 [−0.6, +1.3]−2.8 [−4.1, −1.6]
Pooled, reasoning-heavy0.0 [−1.5, +1.5]−7.8 [−9.9, −5.7]
  • Pooled. IF-FT is non-inferior over all seven domains at δ = 1. Over the three reasoning-heavy domains alone the test is inconclusive at δ = 1, but the interval rules out losses larger than 1.5 points, against −7.8 [−9.9, −5.7] for FP-FT. FP-FT is significantly negative in both pools.
  • Per domain. No domain is significantly negative. Math and health are non-inferior at δ = 2. The other five are inconclusive, with lower bounds from −2.1 to −3.5.
  • Reruns. Averaging three seeds per question gives intervals of similar width: math −0.3 [−2.3, +1.7], physics −0.3 [−2.7, +2.2], history +0.7 [−1.8, +3.3]. The width is set by the number of test questions (257–1,319), not by seed variance. We therefore base the claim on the pooled test.

The revised §3.1 defines lossless reinsertion as paired non-inferiority at these margins, and Appendix K reports all paired CIs. We also correct the comparison with FP-FT. Per domain, the FP-FT gap is significant only on GSM8K. After reinsertion, IF-FT exceeds FP-FT by 10.0 [+7.7, +12.3] on GSM8K and by 5.1 [+3.6, +6.6] pooled over the reasoning-heavy domains, while physics (+3.1 [+0.0, +6.6]) is at the edge of significance and math (+2.1 [−0.2, +4.7]) is not significant on its own. The revised §1 and §4.3.2 state it this way.

Q3: Composition mechanism

  1. Scope of the claim. Yes, the claim holds only for parameter overwriting. The revised §3.4 reads: "Parameter conflicts arise only in shared slots, although updates in disjoint slots can still interact through the shared computation."
  2. Interaction through disjoint slots. With module a's parameters held fixed, adding the partner's non-shared experts changes accuracy by at most 0.7 points, and no interval excludes zero. Averaging the shared slots costs 1.8–2.6 points in high-overlap pairs.
  3. Fixed-pair overlap control. We ran the control you suggested. With math fixed and history as the partner, raising the shared slots from 99 to 347 lowers math's composed accuracy by 2.1 [−4.2, 0.0] points. This is consistent with an effect of overlap beyond domain identity, although the interval reaches zero and the intervention also changes which experts are shared. The slot count alone is not sufficient, however: a physics subset with half the natural overlap, but sharing the slots both domains use most, lowered math's retention further (0.27 vs 0.62).
  4. Low-overlap pairs without history. Pairs with law (ω = 0.32–0.34) retain 0.67–0.77 of the gain, more than the same modules retain with high-overlap partners (0.58–0.65), but within the CIs.
  5. Restricted claim. The revised §4.3.4 reports overlap "as an indicator associated with retention, not as an established cause".

Q4: Non-target performance and applicability

Per-domain changes. We computed paired CIs for every other-domain cell of the seven IF-FT modules (28 cells). No cell is significantly negative. Three are significantly positive: math → physics +2.7 [+0.4, +5.4], health → psychology +2.6 [+0.9, +4.5], and psychology → health +1.5 [+0.2, +3.0]. The MMLU-math change after GSM8K training is −0.7 [−3.3, +1.9], which is not significant. Appendix K lists all 28 cells, and the revised §4.3.2 states that none is significantly negative.

Non-inferiority vs improvement. We agree that these must be kept apart. "Lossless" refers only to the reinsertion gap, relative to the trained module. The gain over the untrained full model is a separate quantity, and the abstract, §4.3.5, and Appendix I report it separately for the non-modular models. The abstract states: "In a standard MoE trained on the same data, reinsertion is equally lossless, but even after training the extracted module falls below its own full model."

  • Standard MoE, math. Reinsertion is lossless (37.5 vs 34.7 for the module), and the reinserted model is 1.9 points above its full model. The module itself, however, is 0.9 points below the full model, and whole-model fine-tuning improves the model more (40.0 vs 37.5), whereas on EMO the two tie (41.7). Lossless reinsertion and a module worth training apart are therefore separate properties: freezing the interface gives the first on both models, while the second depends on modularity.
  • Qwen3-30B-A3B (Appendix I). Reinsertion is lossless relative to the module (+10.4). The reinserted model is still 8.6 points below the untrained full model.
Reviewer_Nm2pOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Paper reviews methods for memory-efficient fine-tuning of MoE models, in particular those that are modular, i.e., where certain domains can be handled well by a subset of the experts, using a fraction of the total parameters. The core problem and contribution is how to FT modular subsets of experts such that the tuned models can be re-inserted into the overall MoE without (1) suffering performance loss on their specialized domain and (2) degrading performance of other domains. The paper teases apart what causes degradation in the re-inserted scenario, attributed it to the non-expert component parameters (called the "interface") having become suboptimal with the FTed experts. As a solution, they propose freezing the interface components during expert FT, and show that that essentially preserves (up to training noise) the target -domain performance, while also somewhat helping the non-target domain performance. They also show that as long as the expert subsets associated with different don't overlap too much, one can re-insert multiple FTed experts and largely preserve the FT benefit in the unified model (they discuss different strategies for reconciling expert components that contribute to multiple domains).

Critical strengths

The paper provides an interesting analysis of problems and recommendations for how to best FT modular components within MoE LLMs. I think it is worthwhile reading and should be published since it gives both increased understanding of how MoE training operates and how to best reduce the memory footprint of modular fine-tuning.

  • The paper has theoretical insight.
  • The paper give valuable recommendations for practical LLM applications.
  • The paper is also a good survey of different parameter-efficient and modular training strategies for MoEs.
Critical weaknesses

No critical weaknesses, other than in the presentation.

The principles behind the design / learning modular structure in MoE should be summarized for the benefit of the reader who is not depe into this field. The reference to EMO is useful but the essentials could be conveyed in section 2 or 3, since achieving modularity is critical for the present work to be applicable.

Figure 1 is confusing because critical difference between architecture (c) and (d) is not visualized. In both diagrams the "interface" components are shown as frozen. The best approach would be to COLOR-code what is frozen during FT vs. what is frozen between the original model and the final model used at inference time. So you need two kinds of "frozen" in the visual depiction.

Questions for authors

The presentation is quite clear, except the issues having to do with clarify of presentation, repeated here:

The principles behind the design / learning modular structure in MoE should be summarized for the benefit of the reader who is not depe into this field. The reference to EMO is useful but the essentials could be conveyed in section 2 or 3, since achieving modularity is critical for the present work to be applicable.

Figure 1 is confusing because critical difference between architecture (c) and (d) is not visualized. In both diagrams the "interface" components are shown as frozen. The best approach would be to COLOR-code what is frozen during FT vs. what is frozen between the original model and the final model used at inference time. So you need two kinds of "frozen" in the visual depiction.

2 comments

We thank Reviewer Nm2p for their positive and constructive feedback! We especially appreciate their recognition that the paper "is worthwhile reading and should be published since it gives both increased understanding of how MoE training operates and how to best reduce the memory footprint of modular fine-tuning," that it "has theoretical insight," and that it gives "valuable recommendations for practical LLM applications." We also value their view that the paper "is also a good survey of different parameter-efficient and modular training strategies for MoEs."

Below, we respond to both presentation points, and we would be glad to address any further questions during the discussion period!

Background on how modularity is obtained

We agree that this background is essential, because our method applies only to modular MoEs. The revision adds the mechanism to the first paragraph of §2:

"EMO routes all tokens of a pretraining document within one pool of experts, chosen by the tokens' average routing probabilities, so experts group by domain and a domain's top 12.5% of experts remain usable on their own."

In more detail:

  • Mechanism. For each pretraining document, EMO averages the routing probabilities over the document's tokens and takes the top experts as the document's pool. Every token then routes only within this pool, and the shared expert stays active. Different documents use different pools, and the pool size is sampled per document.
  • Effect. Because a document usually stays within one domain, experts that are used together come to specialize by domain. Keeping 25% (12.5%) of the experts costs EMO about one (three) points, whereas a standard MoE breaks under the same restriction.
  • Link to our method. Our subset selection (§3.1) uses the same ranking, average routing probability, on a domain's calibration data. We call an MoE modular if, for each domain, such an untrained subset keeps most of the full model's accuracy. EMO's 32-expert subset retains 93.5% of the full model's accuracy on math and 101% on GSM8K. The standard MoE's subset retains 79.5% and 17%.
  • Consequence. §4.3.5 shows what this property changes: reinsertion is lossless on both models, but only on the modular one is the module worth training apart.

Figure 1: two kinds of "frozen"

Thank you for this suggestion. The revised Figure 1 follows it. One clarification first: ESFT and IF-FT update the same kind of parameters, namely routed experts only. Panels (c) and (d) therefore share the same "frozen during training" coding. They differ in three ways that the original figure did not show:

  1. Memory. ESFT keeps the full 14B model in memory. IF-FT holds only the extracted 3.9B module.
  2. Routing during training. ESFT routes among all 127 routed experts. IF-FT routes among the module's 31 per layer.
  3. Output. IF-FT produces a standalone module whose experts are written back to their original slots, so independently trained modules can be composed.

The revised figure encodes these with separate visual channels. Fill color marks parameters as trainable or frozen during training. An orange outline marks parameters that differ in the final model. A dashed gray box marks parameters that are not in memory during training. Panel (d) now shows the full model outside memory, the module routing over 31 experts, reinsertion into the original slots, and composition with a second module. The caption states the three differences above.

Dear Reviewer Nm2p,

Thank you again for your positive review. Following your suggestions, the revised PDF adds how EMO obtains modularity to §2, and Figure 1 now separately encodes which parameters are trained during fine-tuning and which change in the final model. We would be glad to hear whether these changes address your presentation concerns, and to answer any further questions before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful feedback. Below, we summarize the contribution, the evidence, and the changes in the revised PDF.

The question. Modular pretraining enables MoE language models such as EMO to form domain expert subsets that work on their own at inference time. Such a subset is not necessarily one that can be updated on its own. Expert selection decides which parameters to update. We ask whether the improvement learned by a separately trained subset survives when only its experts are written back into the full model, and whether such subsets can be combined.

Evidence.

  • Full-parameter module fine-tuning loses much of its gain at reinsertion. Restoring the fine-tuned interface, freezing it during training, and the new selective-freezing experiment (Appendix L) all point to co-adaptation between the experts and the interface. Reviewer xSUx confirmed that the selective-freezing results address their concern.
  • IF-FT makes reinsertion lossless pooled over seven domains (+0.3 [−0.6, +1.3], margin 1 point), and no domain shows a significant loss (Appendix K).
  • At matched capacity, IF-FT is not more accurate than training the same experts inside the full model. Its benefit is training without the full model in memory (peak memory 32.8 vs 52.1 GB) and a standalone module.
  • Modules compose by slot without further training. The partner's non-shared experts change accuracy by at most 0.7 points, and we report the relation between overlap and retention as an association.

Changes in the revised PDF (in blue).

  • Paired non-inferiority and CIs (§3.1, §4.3.2, Appendix K), and selective freezing (§3.3, Appendix L).
  • Narrowed interference wording (§2, §3.4, §4.3.4), and measured cost on one B200 GPU (§4.3.3, Figure 4, Appendix G).
  • ESFT attribution and the LoRA tuning budget (§4.3.3, §4.3.4, Appendix E), and EMO background with a new Figure 1 (§2).
  • A precise statement on other domains in the abstract, a corrected error in Appendix J, and split semicolon-joined sentences.

We also state three points more precisely than in our earlier responses. The overlap control is consistent with, but does not establish, an effect beyond domain identity. The selective-freezing result refines the principle behind IF-FT rather than identifying a minimal frozen set. We no longer attribute the standard MoE's gain to its untouched experts (§4.3.5).

The composition analyses and the selection × training-location comparison will be added as an appendix in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.