acceptodds
Under review as a conference paper at ICLR 2027

LLM2llm: Parameter-Efficient Adaptation via Transferred Experts from MoE LLMs

Abstract

Stronger language models acquire rich knowledge through expensive pre-training, but transferring it typically requires learning from teacher-generated supervision. We propose LLM2llm, a framework that extracts reusable expert components from a stronger source Mixture-of-Experts model for direct inheritance by target pretrained model. LLM2llm identifies cross-domain generalist experts using multi-domain routing statistics and condenses them through low-rank subspace extraction and score-weighted fusion. This one-time extraction requires no additional training, and the resulting experts can be reused across multiple downstream tasks. Injected as residual branches, only the transferred experts are updated during downstream supervised fine-tuning; the original target parameters remain frozen, and the source model is no longer required. Across two heterogeneous source–target settings, LLM2llm achieves higher average performance than Full Fine-Tuning and representative parameter-efficient baselines on language understanding and dialogue generation tasks. Ablations support the effectiveness of expert selection, condensation, and transferred initialization. These results demonstrate the potential of directly inheriting reusable model components to improve downstream learning efficiently.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large language models, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_PMuLOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose LLM2LLM, a parameter-efficient adaptation framework that aims to transfer knowledge from a larger Mixture-of-Experts (MoE) model to a smaller pre-trained model, without relying on teacher-generated supervised learning. The core hypothesis is that experts consistently activated across multiple domains contain generalizable knowledge that can be reused by other models.

LLM2LLM consists of three stages. First, it identify domain-generalizable experts using routing statistics based on coverage, robustness, and entropy. Second, it combine the selected experts through channel alignment, SVD-based shared subspace extraction, and score-weighted fusion. Third, to accommodate differing hidden dimensions, it use fixed random projections to inject the resulting experts into the pre-trained target model as residual branches. During the subsequent fine-tuning process, the original target model remains frozen, and only the injected experts are updated.

The authors evaluated LLM2LLM using two source-target pairs: Qwen3.5-122B to OLMoE-7B, and Qwen3.5-35B to LLaMA2-7B. LLM2LLM achieved higher average scores on benchmarks for language understanding and conversational generation compared to Full Fine-Tuning, LoRA, PiSSA, and ESFT, while updating only 0.54% and 0.19% of the target model's parameters, respectively. Ablation studies confirm that source-based initialization, expert selection, and condensation all contribute to the observed performance improvements. Overall, these experiments demonstrate that initializing additional adaptation modules using selected and integrated source-expert parameters can enhance subsequent performance. Whether this improvement reflects the identification and transfer of generalizable knowledge, or simply the benefits of using pre-trained parameter initializations, remains an important distinction.

Critical strengths
  1. Multi-criteria approach for selecting reusable experts.

The strategy for selecting experts is persuasive. Rather than relying solely on average routing frequency, the authors evaluate cross-domain generality from three complementary perspectives. First, 'coverage; measures the breadth of an expert’s activation across different domains. Second, 'robustness' captures whether an expert remains active even in less favorable domains. Third, 'entropy' assesses how evenly distributed an expert’s routing activity is. These criteria provide a valid method for distinguishing between experts that are activated broadly and consistently, and those confined to only a few domains. Using the selected experts to construct source-derived initialization weights is a practical and well-motivated design choice.

  1. Promising empirical results for source-based expert initialization and parameter-efficient adaptation.

This method outperforms reported fine-tuning baselines while updating only 0.19% to 0.54% of the target parameters. Moreover, an ablation study reveals that replacing the transferred expert modules with randomly initialized modules of the same architecture and size causes the average accuracy to drop from 75.30 to 52.46. This significant performance gap demonstrates that pre-trained source expert parameters provide a meaningful initialization that goes beyond merely increasing learning capacity.

Critical weaknesses

The central idea of identifying broadly activated experts from a pre-trained MoE, consolidating their parameters, and transferring them to another model has already been explored by XPERT. Furthermore, prior expert-merging methods such as M-SMoE have investigated routing-guided expert selection, channel alignment, weighted merging, and low-rank compression. While LLM2LLM introduces different selection criteria and condensation procedures, the paper does not sufficiently explain what is fundamentally new about these components. Its clearest distinction appears to be injecting source-derived experts into an already pre-trained target model for parameter-efficient adaptation. However, the paper does not compare its extraction procedure against XPERT-based alternatives under the same adaptation framework. This makes it difficult to determine whether the proposed selection and condensation methods constitute substantial methodological contributions beyond existing techniques.

The paper assumes that experts consistently activated across domains contain generalizable knowledge and that dominant SVD directions capture shared knowledge while suppressing expert-specific variation. Although the proposed selection criteria are reasonable, neither assumption is directly validated. More importantly, the ablation results suggest that much of the improvement may come from pre-trained expert initialization rather than the proposed knowledge selection and condensation mechanisms. Random expert selection already achieves 71.30 average accuracy compared with 75.30 for LLM2LLM, and the gains from both selection and condensation are heavily concentrated on WinoGrande. Without stronger controls, such as repeated random selections, direct analysis of shared knowledge representations, and evaluation before downstream fine-tuning, it remains unclear whether the method genuinely transfers generalizable knowledge or primarily provides a favorable initialization for additional trainable FFN modules. This distinction is central to the paper's main scientific claim.

Questions for authors
  1. Section 3.1 assumes that experts consistently routed across domains contain generalizable knowledge. However, 'Probing Semantic Routing in Large Mixture-of-Expert Models' shows that routing depends on semantic similarity, while 'Decoding Knowledge Attribution in Mixture-of-Experts' suggests that different knowledge relations may activate distinct neurons within the same expert. Have the authors measured whether their selected generalist experts actually use overlapping neurons or activation subspaces across domains? Could their high routing scores instead reflect shared semantic features or domain-specific computations within the same expert?

  2. In Table 3, Random Selection already achieves 71.30 accuracy compared with 75.30 for LLM2LLM. Moreover, excluding WinoGrande reduces the average improvement from 4.00 to 1.80 points. How stable are these differences across multiple random expert selections and training seeds? Does the proposed generalist score correlate with downstream transfer performance across experts?

  3. Appendix C assumes that recurring parameter structures dominate the leading SVD directions, while expert-specific variations contribute weaker or unstructured components. Have the authors measured whether the retained directions are actually shared across domains? Can they compare the functional contributions of retained and discarded directions to determine whether SVD preserves generalizable knowledge rather than simply high-energy parameters?

  4. In Table 3, Direct Fusion achieves 72.38 compared with 75.30 for LLM2LLM, but the improvement is concentrated on WinoGrande (+10.81 points), with an average gain of only 0.95 points on the other four benchmarks. Can the authors separately ablate channel alignment, score weighting, and SVD projection to identify which component produces the improvement?

  5. Random Initialization performs substantially worse than source-derived initialization, but this does not distinguish knowledge transfer from favorable pretrained initialization. Can the authors compare against an intact pretrained source expert or a pretrained target FFN injected using the same residual architecture? Additionally, what performance improvement is observed immediately after injection, before downstream SFT?

  6. Equation (6) uses a fixed projection generated by QR decomposition of a random Gaussian matrix. How much does performance vary across different projection seeds? Since this projection matches dimensions without explicitly aligning semantic representations, how does its performance compare with a learned or activation-based projection under the same training budget?

  7. Appendix A reports approximately 50K calibration tokens per domain from GitHub, arXiv, Wikipedia, C4, and B3G. What are the exact domain definitions, and what values are used for the selection thresholds and weights in Equation (2)? How consistent are the selected expert sets when the calibration data size or domain composition changes?

  8. The experiments use two source-target pairs, with both targets being approximately 7B models. Can the same condensed experts from one source be evaluated on multiple target architectures or sizes, particularly smaller 1B-3B models? This would help establish whether the extracted components are genuinely reusable across models rather than effective only for the tested pairs.

  9. Figure 3 attributes the weaker Full FT results to overfitting, despite its substantially lower training loss. Were learning rates, regularization, and checkpoint selection tuned independently for each baseline? Can the authors report validation curves and multi-seed results to distinguish overfitting from suboptimal optimization settings?

  10. For generation tasks, can the authors supplement ROUGE-L with factual accuracy or instruction-following evaluations to assess whether the improvements reflect meaningful capability gains?

6 comments
AuthorsOpenReview ↗

Response to Reviewer PMuL [1/3]

Response to Reviewer #PMuL

Thank you for your careful reading and insightful questions. Thank you for the time and care you devoted to reviewing our paper. Your constructive feedback helped us strengthen our evaluation and clarify what our results support.

Q1 Novelty and relation to prior work. (Weakness 1)

R1 We thank the reviewer for asking us to clarify how LLM2llm relates to XPERT and M-SMoE. Our central contribution is a method for adapting an existing pretrained model by transferring expert weights from a separate source model into trainable residual branches.

  1. Relation to XPERT. Both methods extract and consolidate source expert weights for reuse. XPERT selects experts using mean routing activity and its variation across domains, consolidates their weights with Tucker decomposition, and uses the resulting weights to initialize the FFNs of a new model before pretraining. LLM2llm selects experts using cross-domain coverage, lower-quartile activity, and routing entropy. It aligns intermediate channels and estimates a shared subspace for each projection using joint SVD, then projects the score-weighted average onto that subspace.
  2. Relation to M-SMoE. M-SMoE aligns and groups experts within an MoE and merges them using activation-frequency weights. Its extension, MC-SMoE, further compresses the merged experts. LLM2llm uses cross-domain generalist scores for selection and weighting, and uses the resulting condensed weights to initialize trainable residual branches in a separate, already pretrained target model.
  3. Main contribution and scope of the evidence. Our central contribution is a parameter-efficient way to adapt an existing model using expert weights from a separate source model. The target backbone stays frozen, and downstream training updates only the injected experts. Table 3 evaluates the contribution of our selection and condensation choices within this framework through comparisons with Random Selection and Direct Fusion. We will clarify this scope when discussing the empirical evidence for these components.

Q2. Cross-domain expert selection and downstream transfer. (Questions 1, 2, and 7 & Weakness 2)

R2

R2(a). Generalist scoring and downstream transfer (Questions 1 and 2 & Weakness 2). The generalist score measures how broadly and consistently an expert is routed across calibration domains. It combines ​domain coverage, lower-quartile routing activity, and routing entropy​. We use these signals to identify candidates for cross-domain transfer.

In Table 3, Generalist Selection achieves 75.30% average accuracy across five tasks, compared with 71.30% for Random Selection. Excluding WinoGrande, the average gain is ​1.80 percentage points​. To assess sensitivity to the choice of random experts, we sampled three expert sets with different seeds, holding the number of experts per layer, condensation procedure, projection, and SFT seed fixed.

Expert selectionWinoGrande Accuracy (%)PIQA Accuracy (%)
Random Selection (3 draws, mean ± SD)51.43 ± 0.2183.95 ± 0.18
Generalist Selection64.0186.45

Generalist Selection outperformed the mean of the three random draws by 12.58 percentage points on WinoGrande and ​2.50 points on PIQA​. These results support the downstream benefit of the generalist selection rule at the level of the selected expert sets.

R2(b). Calibration domains and sample size (Question 7). We drew on OLMo’s open pretraining corpus when choosing GitHub, arXiv, Wikipedia, C4, and B3G as calibration sources. These sources cover different types of text and subject matter. We sampled ​20,000 tokens per domain, for a total of 100,000 tokens​, to reduce the influence of individual documents on the routing statistics. We will correct the per-domain token count in Appendix A.

AuthorsOpenReview ↗

Response to Reviewer PMuL [2/3]

(continued from [1/3])

Q3. Shared-subspace condensation and component ablations. (Questions 3 and 4 & Weakness 2)

R3

R3(a). What the retained directions represent (Question 3). Experts selected for broad cross-domain activity can still retain structure specific to individual experts, domains, or tasks. Selection identifies candidate experts, and condensation further refines what we transfer from them. After channel alignment, joint SVD estimates a shared low-rank basis that minimizes reconstruction error across the selected experts’ weight matrices. We project the score-weighted average onto this basis to retain dominant directions estimated jointly across experts and suppress components outside the retained subspace.

Appendix C motivates the hypothesis that ​recurring parameter structure is concentrated in these leading directions​. Our analysis characterizes these directions in weight space. Whether they are used across domains remains an empirical question.

R3(b). Downstream effects of the design choices (Question 4 & Weakness 2). Direct Fusion starts from the ​same score-selected experts​, retains ​channel alignment​, and uses the same injection coefficient (\beta) as LLM2llm. It then averages the aligned weights uniformly in the original parameter space.

VariantChannel alignmentExpert selectionSVD projectionWinoGrande Acc. (%)PIQA Acc. (%)
LLM2llmYesGeneralist scoreYes64.0186.45
Without channel alignmentNoGeneralist scoreYes53.3885.53
Random SelectionYesRandomYes51.2284.11
Direct Fusion (uniform averaging)YesGeneralist scoreNo53.2085.20

Channel alignment puts intermediate channels into correspondence before fusion. Removing it lowers accuracy by 10.63 points on WinoGrande and ​0.92 on PIQA​. Generalist-score selection favors experts with broad and consistent routing activity. Replacing it with random selection lowers accuracy by 12.79 and ​2.34 points​, respectively.

​Shared-subspace condensation improves on Direct Fusion across all five tasks in Table 3​, including +1.25 points on PIQA and ​**+10.81 points on WinoGrande​. Starting from the same selected and aligned experts, these gains support the value of refining their aggregation before transfer. The comparison captures the ​combined contribution of SVD projection and score-weighted fusion**​.

Q4 Pretrained initialization and performance before SFT. (Question 5 & Weakness 2)

R4

Thank you for suggesting a comparison with pretrained initializations. We use the same residual branch architecture, injection layers, trainable parameter count, and SFT budget across variants. For ​Intact Source Expert​, we directly copy the highest-scoring Qwen expert from the selected set at each injection layer. For ​Target-native Expert Init​, we initialize each branch with the most frequently routed expert from the corresponding OLMoE layer on the calibration data. We match the initial residual output RMS of the pretrained variants to that of LLM2llm on the same calibration samples.

Branch initializationWinoGrande Acc. (%) after SFTPIQA Acc. (%) after SFT
Random Init51.1449.46
Intact Source Expert57.5385.09
Target-native Expert Init49.5779.30
LLM2llm64.0186.45

LLM2llm outperforms Intact Source Expert by 6.48 and ​1.36 percentage pointson WinoGrande and PIQA, respectively. Both variants use experts from the same Qwen checkpoint, suggesting that condensing multiple selected experts provides a more effective initialization than transferring the highest-scoring expert alone. One possible explanation is that the selected experts contribute complementary information across domains. LLM2llm also improves over Target-native Expert Init by 14.44 and 7.15 percentage points​, showing that the full transfer pipeline offers a benefit over initializing the same branch with the target model’s own pretrained expert.

Does expert injection improve accuracy before any downstream SFT?

Under our strict answer-matching protocol, accuracy was near zero both for the base models and immediately after expert injection. Since this metric also depends on whether the model follows the requested answer format, the pre-SFT scores provide limited evidence about which capabilities the injected experts carry. The measurable benefit appears during downstream adaptation. Under the same SFT budget, the transferred experts provide a stronger initialization that leads to higher final task accuracy. Our results therefore support the use of transferred experts for supervised adaptation, while immediate transfer of behaviors acquired during source-model post-training, such as instruction following, remains unestablished.

AuthorsOpenReview ↗

Response to Reviewer PMuL [3/3]

(continued from [2/3])

Q5 Sensitivity to projection seeds and comparison with a learned projection. (Question 6)

R5 The fixed QR projection in Eq. (6) maps between the source and target hidden dimensions. During SFT, the transferred expert weights adapt to the target through this fixed mapping. To measure sensitivity to the projection, we evaluate three QR seeds while keeping the condensed experts, SFT seed, and training configuration fixed. For the learned-projection comparison, we use one of these frozen runs as the reference and initialize the trainable projection with the exact same QR matrix. Both variants update the expert weights using the same SFT data and number of optimization steps.

ProjectionTrainable parametersWinoGrande Acc. (%)PIQA Acc. (%)
Fixed QR, 3 seeds (mean ± SD)37.8M64.14 ± 0.1186.39 ± 0.18
Fixed QR, paired reference37.8M64.0186.45
Trainable projection, same QR initialization62.97M60.3784.72

The fixed projection shows ​low sensitivity to the three tested seeds​, with standard deviations of 0.11 and 0.18 percentage points on WinoGrande and PIQA, respectively. In the paired comparison, training the projection ​reduced accuracy by 3.64 points on WinoGrande and 1.73 points on PIQA​, while adding 25.17M trainable parameters.

One possible explanation is that freezing the projection gives the pretrained experts a stable mapping through which to adapt to the target. When the projection is trainable, SFT must also learn that mapping from a random QR initialization, jointly with adapting the expert weights. Under the same data and step budget, this joint optimization may be harder than adapting the expert weights through a fixed mapping. The paired result favors the frozen projection under the tested SFT budget.

Q6. Reusing the same condensed experts across target models. (Question 8)

R6 Our experiments already cover both a MoE target (OLMoE-7B) and a ​dense target (LLaMA2-7B). In both cases, the Qwen source and the target belong to different model families. The condensed experts are injected as residual branches, and a target-specific projection handles differences in hidden width. This allows the source and target to differ in size, model family, and architecture.

Expert extraction and condensation are performed ​once per source checkpoint​. The resulting expert weights can serve as a shared initialization for different tasks on the same target and for targets of different sizes or architectures. For each new target, we configure the injection layers and projection, then fine-tune a separate copy of the same condensed weights.

Our two experiments demonstrate transfer to both dense and MoE targets using separately extracted expert sets. Reusing the exact same condensed weights on a 1B–3B target would directly test how well this reuse works at a smaller scale.

Q7. Full fine-tuning and model selection (Question 9)

R7. In the original OLMoE-7B experiments on Dolly, Full FT and LLM2llm used a shared learning rate of 2e-5 and were evaluated at epoch 10. In the follow-up, we tested 1e-5 and 2e-5 for Full FT at fixed weight decay. We selected the Full FT learning rate and both methods’ checkpoints by ROUGE-L on a held-out Dolly validation split, then evaluated the selected checkpoints on DollyEval.

MethodLearning rateCheckpointValidation ROUGE-LDollyEval ROUGE-L
Full FT2e-5Epoch 10 → Epoch 6 (best on validation)22.80 → 23.2022.58 → 22.65
Full FT1e-5Epoch 8 (best on validation)22.6522.10
LLM2llm2e-5Epoch 10 (best on validation)31.8031.75

Validation favored the original 2e-5 learning rate for Full FT. Selecting its epoch-6 checkpoint improved DollyEval ROUGE-L by ​0.07 points​. With both methods using validation-selected checkpoints, ​LLM2llm retained a 9.10-point advantage​.

Figure 3 shows that Full FT’s training loss continued to fall while DollyEval ROUGE-L largely plateaued. The follow-up shows a modest late-epoch decline in validation ROUGE-L, with little change on DollyEval when selecting the earlier checkpoint. We will present overfitting as a possible explanation for the late-training behavior.

Q8 What do the generation results measure? (Question 10)

R8 Generation evaluation. Our generation experiments report ROUGE-L, which measures overlap with the reference responses. The observed gains therefore indicate closer agreement with those references. We will clarify this scope in the revision. Factual accuracy and instruction following would require separate evaluations.

Thank you again for your thoughtful review. We wish you all the best in your research.

Reviewer_1NKTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LLM2llm, a parameter-efficient adaptation framework that transfers expert parameters from a stronger MoE model to a smaller pretrained target model. The framework has three training-free stages: (1) identifying cross-domain generalist experts via multi-domain routing statistics; (2) condensing the selected experts through channel alignment, shared low-rank subspace extraction (SVD), and score-weighted fusion; and (3) injecting the condensed experts as trainable residual branches into the frozen target backbone. Experiments on two heterogeneous source–target pairs (Qwen3.5-122B→OLMoE-7B, Qwen3.5-35B→LLaMA2-7B) report improvements over Full FT, LoRA, PiSSA, and ESFT while updating only 0.54% / 0.19% of target parameters.

The core idea of directly inheriting reusable expert components rather than learning from teacher-generated supervision is appealing, and the extracted experts can be reused across tasks without the source model. However, my assessment is limited by concerns about the reliability of several baselines, the strength of the evidence attributing gains to learned source structure, and the delineation from closely related prior work.

Critical strengths

Meaningful Problem and Coherent Framework. Direct parameter-level reuse of stronger pretrained models is a valuable direction, and adapting a pretrained target without teacher-generated supervision is practically relevant. Expert selection, condensation, and injection form a logically connected pipeline: channel alignment addresses permutation ambiguity in parameter fusion, and the appendix provides a parameter-space justification under explicit assumptions.

Comprehensive Empirical Evaluation. The experiments cover both dense and MoE targets and diverse understanding and generation tasks, reporting substantial improvements over representative baselines with a very small trainable parameter fraction. The random initialization, random selection, and direct fusion ablations provide useful evidence; in particular, the random-init comparison supports the usefulness of transferred initialization beyond simply adding trainable capacity under the evaluated configuration.

Critical weaknesses

Unusually Low PEFT Baselines. In Table 1, LoRA-r8/r16 and PiSSA on LLaMA2-7B reach HellaSwag accuracies of 25.16 / 24.28 / 24.68, close to random guessing on a four-choice task; even Full FT (60.73) sits well below typical zero-shot LLaMA2-7B performance, suggesting a possible evaluation-protocol or SFT-data issue rather than merely mistuned PEFT baselines. Identical optimization settings do not guarantee fairness across adaptation methods, and the main conclusions depend on the reliability of these comparisons.

Attribution of Gains to Learned Structure. The random-init ablation does not separate learned source-model structure from weight-scale, spectral, or branch-output-magnitude effects that may themselves aid optimization. One controlled comparison would substantially strengthen the claim: target-native expert initialization on OLMoE (matching parameter budget and branch output scale), or an initialization preserving singular-value spectra while randomizing directions. I do not regard both as necessary.

Positioning and Condensation Details. XPERT (arXiv:2605.08842) is closely related in cross-domain expert selection, decomposition-based consolidation, and parameter-level reuse; the key distinction—XPERT initializes a new model's FFN while LLM2llm adds residual branches to a frozen pretrained backbone—should be made explicit in the novelty discussion. Additionally, it is unclear whether the Direct Fusion baseline retains channel alignment before averaging; controls such as SVD with uniform weights and score-weighted fusion without SVD would help separate the condensation components, though their absence alone would not determine my recommendation.

Questions for authors

Q1. Please report the target models' performance before downstream fine-tuning, clarify the training and answer-scoring protocols, and explain the unexpectedly low HellaSwag results. What hyperparameter search and model-selection procedures were used for the LoRA/PiSSA baselines?

Q2. Can the authors provide a controlled comparison (target-native expert init or spectrum-preserving randomized init) to support the interpretation that gains come from learned expert structure?

Q3. Which components are shared with XPERT and which are newly introduced? Also, does the Direct Fusion baseline retain channel alignment before averaging?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1NKT [1/2]

Response to Reviewer #1NKT

Thank you for your careful reading and insightful questions. Thank you for the time and care you devoted to reviewing our paper. Your constructive feedback helped us strengthen our evaluation and clarify what our results support.

Q1 Baseline reliability and evaluation protocol. (Question 1 & Weakness 1)

R1

Q1(a). HellaSwag training and evaluation protocol. Thank you for drawing our attention to the unusually low HellaSwag scores. All adaptation methods on LLaMA2-7B used the ​same HellaSwag SFT examples​. At evaluation, we prompted each model to generate exactly one answer letter and used ​strict exact-match accuracy (Acc.)​. For example, when the gold answer is B, The answer is B. receives zero credit. Under this protocol, both pretrained base checkpoints, ​LLaMA2-7B and OLMoE-7B, achieved near-zero HellaSwag accuracy before task-specific SFT​. The metric captures both answer correctness and adherence to the requested format. Published zero-shot results based on candidate likelihood score the four candidate continuations directly.

The near-chance LoRA/PiSSA results on HellaSwag are backbone-dependent in our experiments. Under the same scoring rule and shared training schedule, the LoRA and PiSSA baselines achieved ​79.39–88.03% HellaSwag accuracy on OLMoE-7B​, compared with ​roughly 25% accuracy on LLaMA2-7B​. With LLaMA2-7B held fixed, Full FT and LLM2llm achieved ​60.73% and 70.97% HellaSwag accuracy​, respectively, using the same SFT examples and scoring rule. Together, these comparisons point to the combination of the LLaMA2-7B backbone and the tested LoRA/PiSSA configurations as the likely source of the low accuracy under our protocol. On LLaMA2-7B, all three LoRA/PiSSA variants also trail Full FT on BoolQ, PIQA, and CaseHold.

LLM2llm’s advantage extends beyond HellaSwag. On the same LLaMA2-7B backbone, LLM2llm outperforms all three tested LoRA/PiSSA variants on ​six of the seven tasks​, with 61.12% average accuracy versus 41.05–43.55% for those variants.

Q1(b). Hyperparameters, checkpoint selection, and learning-rate sensitivity. The original Table 1 runs used the listed LoRA/PiSSA configurations, a shared learning rate of 2e-5​​ with cosine decay​, and the ​checkpoint at the end of epoch 3​. We added runs at 5e-5 and 1e-4 on LLaMA2-7B, keeping the SFT data, other hyperparameters, and checkpoint rule fixed. We report all three rates on the same HellaSwag evaluation split.

LLaMA2-7B baseline2e-55e-51e-4
LoRA-r825.1624.7423.96
LoRA-r1624.2823.8623.52
PiSSA24.6824.1723.82

HellaSwag strict exact-match accuracy (%), evaluated at epoch 3.

The original learning rate gave the highest accuracy among the tested rates for all three baselines. LLM2llm’s 70.97% HellaSwag accuracy is 45.81 percentage points above the highest LoRA/PiSSA accuracy in this sweep.

Q2 Controlled initialization ablation. (Question 2 & Weakness 2)

R2 Thank you for suggesting these controls. On OLMoE-7B, Target-native Expert Init initializes the adaptation branches with pretrained experts from the target model. Spectrum-preserving Random Init retains the singular values of each transferred expert weight matrix and replaces its singular vectors with random orthonormal directions. This preserves each matrix’s singular-value spectrum and Frobenius norm while disrupting its learned directions.

Expert initializationTrainable parameters (% of target)WinoGrande Acc. (%)PIQA Acc. (%)
Target-native Expert Init≈37.5M (≈0.54%)49.5779.30
Spectrum-preserving Random Init≈37.8M (0.54%)54.3784.96
LLM2llm≈37.8M (0.54%)64.0186.45

With the same trainable parameter count and per-matrix singular-value spectra, ​LLM2llm outperforms Spectrum-preserving Random Init by 9.64 percentage points on WinoGrande and 1.49 on PIQA​. This supports a contribution from the learned weight directions beyond the spectra and weight norms. Initial branch output magnitude can also depend on how those directions interact with the target activations.

The comparison with ESFT in Table 1 provides complementary evidence. ESFT fine-tunes existing OLMoE experts and achieves 51.07% accuracy on WinoGrande and ​81.34% on PIQA​, updating 1.45% and 1.90% of the target parameters, respectively. LLM2llm updates 0.54% on both tasks and improves accuracy by ​12.94 and 5.11 percentage points​, respectively. Together with the target-native initialization comparison, these results suggest that experts transferred from a stronger source model can provide knowledge that complements the target model’s own experts during downstream adaptation.

AuthorsOpenReview ↗

Response to Reviewer 1NKT [2/2]

(continued from [1/2])

Q3 Relation to XPERT. (Question 3 & Weakness 3)

R3 We thank the reviewer for raising this question. Both XPERT and LLM2llm extract expert knowledge from a source model for reuse. They differ in their ​objectives, expert selection and condensation, and how the resulting experts are used​.

  1. Expert selection. XPERT ranks routed experts by their mean routing activity minus the mean absolute deviation of that activity across domains. LLM2llm first ​requires activity in a minimum fraction of domains​, then ranks eligible experts using ​domain coverage, lower-quartile activity, and routing entropy​.
  2. Expert condensation. XPERT stacks expert weights into a three-way tensor and applies Tucker decomposition along the input, output, and expert dimensions. It averages the factor along the expert dimension to reconstruct a consolidated weight matrix, then uses SVD in a subsequent resizing step. LLM2llm first ​aligns intermediate channels across experts using Hungarian matching​. For each of the gate, up, and down projections, we concatenate the aligned matrices and use ​joint SVD to estimate a shared right-singular subspace​. We then project the score-weighted average onto this subspace to obtain the condensed expert weights. The joint SVD determines which directions are retained, and the generality scores determine how much each expert contributes.
  3. Transfer to the target model. XPERT uses the reconstructed weights to ​initialize the FFN layers of a new model before pretraining​. LLM2llm uses the condensed experts to ​initialize additional residual branches in an already pretrained model​. A fixed projection maps target hidden states into the expert’s input space and maps the expert output back to the target space. Downstream adaptation ​updates only the injected experts while keeping the target backbone frozen​.
  4. Motivation and practical advantage. XPERT uses source expert knowledge to initialize a new model that then undergoes pretraining and supervised fine-tuning. LLM2llm uses that knowledge to ​adapt an existing pretrained base model directly through PEFT​. This provides a direct path from expert transfer to downstream adaptation, and our experiments show ​substantial gains over standard PEFT baselines on several downstream tasks​.

Q4 Does Direct Fusion retain channel alignment? (Question 3 & Weakness 3)

R4 Yes, Direct Fusion retains channel alignment. Direct Fusion retains channel alignment. It uses the same selected experts, injection layers, and injection coefficient β as LLM2llm, and averages the aligned weights in the original parameter space. In Table 3, LLM2llm improves average accuracy from 72.38% to 75.30% across five tasks.

The motivation for joint SVD is that experts active across several domains can still contain domain-specific and expert-specific structure. After channel alignment, we estimate dominant directions jointly across the selected experts. Projecting the fused weights onto this shared subspace retains those directions and attenuates components outside it. The resulting weights form our condensed generalist experts. The improvement over Direct Fusion is consistent with our hypothesis that emphasizing shared parameter directions yields a better initialization for transfer.

Thank you again for your thoughtful review. We wish you all the best in your research.

AuthorsOpenReview ↗

Gentle Follow-up on Rebuttal Responses Before the Discussion Deadline

Dear Reviewer 1NKT,

We would like to sincerely thank you for the time and effort you have devoted to reviewing our paper. We greatly appreciate your thoughtful comments and constructive suggestions, which have been invaluable in helping us improve our work.

We have carefully addressed the concerns raised in your reviews and provided detailed responses and additional evidence during the rebuttal period. As the discussion period is approaching its end, we would be very grateful if you could kindly revisit our responses and share any further thoughts, particularly regarding whether your main concerns have been adequately addressed.

We fully understand your busy schedules and sincerely appreciate the effort involved in the review process. We would be happy to provide any additional clarifications if needed.

Thank you again for your valuable time, consideration, and contributions.

Best regards, The Authors of Paper #1168

Reviewer_3FpZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LLM2llm, which investigates how to directly reuse expert parameters from a stronger MoE model in another pretrained model to improve downstream task adaptation. The method first selects general-purpose experts using routing statistics from multi-domain calibration data, then merges multiple experts from each selected source layer into a single module through channel alignment, shared low-rank subspace extraction, and weighted fusion. These modules are injected into the target model as residual branches via fixed random semi-orthogonal projections, and downstream training updates only the injected expert parameters. The authors evaluate the method on two settings: Qwen3.5-122B→OLMoE-7B and Qwen3.5-35B→LLaMA2-7B. The results show that, while training only 0.54% and 0.19% of the target model’s parameters, respectively, the method achieves better average performance than the reported baselines.

Critical strengths

1, The problem is meaningful, and the design of module reuse is also practically appealing. Extracting experts from the source model into reusable adapter modules is the most commendable aspect of this paper. Once extracted, downstream training no longer requires running the source model or relying on supervision generated by the teacher. If these modules can reliably serve multiple tasks, their one-time preparation cost can be amortized.

2, The experiment covered different target architectures and the main results were quite convincing. The method was applied to both MoE and dense target models, indicating that its implementation does not rely on the original routing mechanism of the target model. On OLMoE, the average language understanding accuracy improved from 69.59 of ESFT to 73.95, while using fewer trainable parameters. This improvement range is worthy of further verification.

3, The author attempts to verify the role of each step of the method rather than merely reporting the final performance. Comparisons such as random initialization, random expert selection, and direct fusion provide preliminary evidence for determining whether the source parameters are useful. The paper also examines the number of experts, retention rank, injection intensity, and injection location.

Critical weaknesses

1, Appendix A mainly adopts a uniform learning rate and training rounds. However, the same hyperparameters do not necessarily mean that all different methods have been fully optimized. The authors attribute the disadvantage of Full FT to overfitting, but they did not provide independent parameter tuning, validation set selection, checkpointing after early stopping, or comparisons. Before these issues are resolved, it is difficult to determine to what extent the reported advantages reflect the benefits of the methods themselves.

2, The improvement of LLM2llm compared to Direct Fusion is notably concentrated in WinoGrande: this improvement amounts to 10.81 percentage points, while the other four items improve by approximately 0.33 to 1.77 percentage points. Random projection, random selection, and downstream training may introduce fluctuations, but the paper does not report the mean and standard deviation of multiple runs. This makes it difficult to judge the stability of the benefits of some components and the best hyperparameters.

3, The paper clearly states that the low-rank projection is only used for initialization, and the complete matrices of the experts are updated during the subsequent training. Therefore, the low-rank condensation itself does not directly reduce the number of trainable parameters in the downstream. The actual cost also includes source model calibration, Hungarian matching, SVD, as well as the additional expert branches and fixed projections during training and inference. Especially on the MoE target model, the injected branches are executed for each token, and the overhead cannot be directly inferred from the proportion of total parameters. Currently, there are no measurements for the time-consuming preparation stage, training memory, throughput, and inference delay. "The non-training preparation stage" is a clear characteristic, but its computational cost still needs to be quantified.

4, The generation experiment requires a clearer definition of the metrics: Table 2 does not directly indicate specific metrics, while Figure 3 uses ROUGE-L. If the main conclusion relies on ROUGE-L, then the evidence is more suitable for supporting improvements in reference text matching, but its support for open-ended dialogue quality and instruction compliance remains limited. Moreover, the key configurations such as thresholds in expert selection, scoring weights, and channel matching weights have not been fully provided.

Questions for authors

1, What initialization distribution was used in Random Init? Does the initial branch output scale match that of the transfer expert? Does the advantage of transfer initialization still hold after separately adjusting β, or using a zero-output initialization of the same scale for the adapter?

2, How sensitive are the results of fixed random projections to the random seed? Can multiple projections and the means and standard deviations under different training seeds be reported, and can the fixed random projections be compared with the learnable projections to help understand the role of cross-model interfaces?

3, Does Direct Fusion retain channel alignment and score weighting? Can the channel alignment, SVD projection, and score weighting be separately removed, and a simple baseline of selecting experts solely based on overall routing frequency be added to determine how much benefit each of the complex filtering and refinement processes brings?

4, Can the complete resource consumption be reported? It is hoped to see the one-time extraction cost, the peak memory and throughput for downstream training, the inference delay, as well as the additional parameter quantity including the fixed projections. If the module is reused for multiple tasks, how does the cost be spread out with the increase in the number of tasks?

5, Could the data division, evaluation metrics, and key hyperparameters be completed? And could the basis for their selection be clarified? Especially, for MMLU, what is the source of the training data, and whether m, r, and β are all selected through an independent validation set. If these configurations remain unchanged on new tasks, will their effects still be stable?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3FpZ [1/3]

Response to Reviewer #3FpZ

Thank you for your careful reading and insightful questions. Thank you for the time and care you devoted to reviewing our paper. Your constructive feedback helped us strengthen our evaluation and clarify what our results support.

Q1 Initialization and projection robustness. (Question 1 & Question 2 & Weakness 2)

R1 Thank you for raising this important concern. R1(a). Random initialization and branch scaling. Thank you for raising these questions. For Random Init (Ablation variant in Table 3 of our manuscript), we initialize the injected expert weights with the PyTorch nn.Linear default, a fan-in uniform distribution on ([−1/din,,1/din])([-1/\sqrt{d_{\mathrm{in}}},,1/\sqrt{d_{\mathrm{in}}}]). Yes, Random Init uses the same injection coefficient (β\beta) as LLM2llm. We hold (β\beta) fixed across all four variants in Table 3, including Random Selection and Direct Fusion, to control for the effect of the injection coefficient. At this shared (β\beta), LLM2llm achieves 75.30% average accuracy versus 52.46% for Random Init, a gain of ​22.84 percentage points​.

R1(b). Projection seed sensitivity and trainable projections. We vary the projection seed and the SFT seed separately, using one shared reference run for a total of five runs. The standard deviations on WinoGrande are 0.11 percentage points across projection seeds and ​0.09 percentage points across training seeds​.

Seed variedProjection seedsSFT seedsWinoGrande Acc. (mean ± SD, %)PIQA Acc. (mean ± SD, %)
Projection42, 43, 4420064.14 ± 0.1186.39 ± 0.18
SFT42100, 200, 30063.95 ± 0.0986.43 ± 0.06

We also compared the frozen projection with a trainable projection initialized from the same QR matrices as the shared reference run. Both variants update the transferred expert weights and use the same SFT data and number of optimization steps.

ProjectionTrainable parametersWinoGrande Acc. (%)PIQA Acc. (%)
Frozen QR (shared reference)37.8M64.0186.45
Trainable projection (same QR initialization)62.97M60.3784.72

Making the projection trainable reduced accuracy by 3.64 percentage points on WinoGrande and ​1.73 on PIQA​, while adding 25.17M trainable parameters. One possible explanation is that a frozen projection gives the transferred experts a stable mapping, allowing their weight updates to compensate for differences between the source and target representations. With a trainable projection, SFT must also optimize the randomly initialized linear mapping while adapting the expert weights. This joint optimization may be harder under the same data and step budget.

Q2 Baseline tuning and checkpoint selection. (Weakness 1)

R2 Thank you for pointing out that a shared training recipe can affect methods differently. On OLMoE-7B, we tested two learning rates for Full FT at fixed weight decay. We used a held-out Dolly validation set to select the learning rate and checkpoint by ROUGE-L, and evaluated the selected checkpoints on DollyEval. LLM2llm retained its original learning rate and used the same checkpoint selection rule.

MethodLearning rateCheckpointValidation ROUGE-LDollyEval ROUGE-L
Full FT, original LR2e-5Epoch 10 → Epoch 6 (best on validation)22.80 → 23.2022.58 → 22.65
Full FT, alternative LR1e-5Epoch 8 (best on validation)22.6522.10
LLM2llm2e-5Epoch 10 (best on validation)31.8031.75

Among the two learning rates tested, validation selected 2e-5 for Full FT, the rate used in the original comparison. Selecting the epoch-6 checkpoint improved Full FT's DollyEval ROUGE-L by 0.07 points over the final checkpoint. With both methods using validation-selected checkpoints, LLM2llm retained a 9.10-point advantage on DollyEval.

Full FT's validation ROUGE-L declined from 23.20 at epoch 6 to 22.80 at epoch 10, consistent with some deterioration late in training. The small DollyEval gain from selecting the earlier checkpoint leaves the broader performance gap largely unexplained. We will revise the discussion of Figure 3 to present overfitting as a possible explanation and report the measured effect of checkpoint selection.

AuthorsOpenReview ↗

Response to Reviewer 3FpZ [2/3]

(continued from [1/3])

Q3 Direct Fusion and component ablations. (Question 3 & Weakness 2)

R3 Thank you for asking us to clarify Direct Fusion. Direct Fusion starts from the same score-selected generalist experts as LLM2llm. Both methods therefore benefit from selecting experts with broad and consistent routing activity across domains. Direct Fusion retains channel alignment and combines these experts by uniform parameter averaging, with no SVD projection. It uses the same injection coefficient (β\beta) as LLM2llm.

VariantChannel alignmentExpert selectionSVD projectionWinoGrande Acc. (%)PIQA Acc. (%)
LLM2llmYesGeneralist scoreYes64.0186.45
Without channel alignmentNoGeneralist scoreYes53.3885.53
Random SelectionYesRandomYes51.2284.11
Direct Fusion (uniform averaging)YesGeneralist scoreNo53.2085.20

Channel alignment. Removing alignment lowers accuracy by 10.63 points on WinoGrande and ​0.92 points on PIQA​. Alignment establishes correspondence between intermediate channels before merging, so similar features can be combined even when they occupy different channel indices.

Expert selection. Replacing generalist-score selection with random selection lowers accuracy by 12.79 points on WinoGrande and ​2.34 points on PIQA​. This comparison supports selecting experts with broad and consistent cross-domain activity. A frequency-only baseline would test how much the cross-domain criteria add beyond aggregate routing frequency.

Expert condensation. Uniform averaging can preserve parameter structure that the selected experts already share. LLM2llm refines this aggregation by using joint SVD to estimate a low-rank subspace from the aligned experts. This subspace serves as a parameter-space proxy for their shared information. We fuse the experts within that subspace, giving greater weight to experts with higher generalist scores. ​Starting from the same selected experts, LLM2llm improves over Direct Fusion on all five tasks in Table 3​, including +1.25 points on PIQA and ​**+10.81 points on WinoGrande**​. These gains measure the combined contribution of SVD projection and score-weighted fusion. Isolating either contribution would require varying it while holding the other fixed.

On WinoGrande, the full LLM2llm configuration also shows low variability in R1(b), with standard deviations of 0.11 points across projection seeds and ​0.09 points across training seeds​.

Q4 Preparation, training, and inference costs. (Question 4 & Weakness 3)

R4 Expert extraction is training-free and runs once per source checkpoint and extraction configuration. For Qwen3.5-122B-A10B → OLMoE-7B, the entire preparation took ​130 minutes​. After extraction, the same expert weights can initialize adaptation branches for target models of different sizes and architectures and for downstream tasks across domains. Each adaptation uses a target-specific projection and its own SFT run. The total extraction cost stays at 130 minutes as more tasks reuse the experts.

StageResourceCost
One-time preparationSource routing calibration40 min
One-time preparationExpert extraction, channel alignment, SVD, and fusion90 min
One-time preparationTotal130 min
Downstream SFTFull FT GPU memory observed during training≈74 GB/GPU
Downstream SFTLLM2llm GPU memory estimate≈25 GB/GPU
Added model weightsTrainable expert weights across four OLMoE layers37.8M
Added model weightsFrozen projection elements across those layers25.17M
Added model weightsTotal≈63.0M

The preparation step and downstream training setup use two 80 GB NVIDIA H100 GPUs. Freezing the backbone removes the need to store its parameter gradients and optimizer states. SVD constructs the initialization, and downstream SFT updates the full injected expert matrices. Both the expert branches and the fixed projections contribute to inference compute. Training throughput and inference latency require separate timing measurements.

AuthorsOpenReview ↗

Response to Reviewer 3FpZ [3/3]

(continued from [2/3])

Q5 Data splits, metrics, and hyperparameters. (Question 5 & Weakness 4)

R5 We thank the reviewer for raising these helpful questions about our evaluation protocol and hyperparameter settings. Metrics and data splits. Table 2 reports ROUGE-L for DollyEval, S-NI, UnNI, SelfInst, and Vicuna. We will state the metric explicitly in the table caption and describe the gains as improved overlap with reference responses. Dialogue quality and instruction following require separate evaluation. For language-understanding tasks such as BoolQ and MMLU, we report Accuracy (%). For MMLU, we use the auxiliary_train split of Hugging Face's cais/mmlu dataset for supervised fine-tuning and the test split for evaluation. For the other language-understanding benchmarks, we likewise use separate training and evaluation splits.

Expert selection and channel alignment. We use an activity threshold of τ = 1.0 and a minimum domain coverage of γ = 0.5. The expert score weights coverage, robustness, and entropy by 0.4, 0.4, and 0.2, respectively. The channel-matching score weights the gate, up, and down projection similarities by 0.4, 0.4, and 0.2, respectively.

Hyperparameter sensitivity. Figures 4 and 5 report results on the same test splits as the main experiments (Table 1 and 2). Across the tasks shown, settings that perform well on one task generally perform well on the others. This indicates that the favorable settings are consistent across the tasks evaluated in the sweeps.

Thank you again for your thoughtful review. We wish you all the best in your research.

Reviewer_c6TxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LLM2llm, a parameter-efficient framework that directly transfers reusable modules from a stronger source Mixture-of-Experts (MoE) LLM into a pretrained target model. The method identifies cross-domain “generalist” experts via multi-domain routing statistics, aligns their channels, condenses them with a shared low-rank subspace (SVD) and score-weighted fusion, and injects the resulting expert branches into the target through fixed projections; only these injected experts are trained during downstream fine-tuning while the target backbone remains frozen. Across two heterogeneous source–target pairs (Qwen3.5-122B→OLMoE-7B and Qwen3.5-35B→LLaMA2-7B), LLM2llm outperforms Full FT and several PEFT baselines on language understanding and dialogue generation with <1% trainable parameters, supported by ablations on expert selection, condensation, and injection design.

Critical strengths
  • Introduces a cross-model transfer mechanism that reuses condensed MoE experts as trainable residual branches in a separate target model without teacher inference or distillation targets.
  • Proposes a principled, training-free pipeline: (i) multi-domain routing-based selection of “generalist” experts, (ii) channel alignment with a Hungarian assignment over combined gate/up/down similarity, and (iii) shared-subspace SVD with score-weighted parameter fusion.
  • Bridges heterogeneous source–target hidden sizes with fixed semi-orthogonal projections, making the approach applicable to both dense and MoE targets.
Critical weaknesses
  • Requires access to the source MoE’s routing decisions and weights; applicability is limited when only black-box access is available or when the source is a dense model.
  • The fixed random projections Pℓ for dimensional bridging are simple but possibly suboptimal; no exploration of learning or sharing small projection adapters that could preserve more structure.
  • The approach assumes channel permutations and shared low-rank structure are sufficiently recoverable across selected experts; potential failure modes (e.g., weak alignment across layers/domains) are not deeply examined.
  • Baseline fairness: Full FT underperforms notably; 3-epoch SFT and uniform hyperparameters may penalize it, and LoRA ranks are small in some settings. ESFT on LLaMA2 is compared using an MoE-converted target rather than the same dense target, which complicates fairness.
  • Missing comparisons to adjacent non-distillation transfer paradigms (e.g., task/weight vectors and training-free merging methods) and to router-only post-training methods, which could be strong alternatives.
  • No reporting of inference overhead (compute/latency/memory) when keeping the injected experts active at test time; practical deployment costs are unclear.
  • Sensitivity to selection hyperparameters (λ weights, τ threshold, γ coverage) and to the choice of source-layer sampling is not studied.
  • Recent MoE compression/merging literature that also uses SVD-based shared subspaces and frequency-weighted fusions (e.g., Sub-MoE, MoE-I2) is not discussed; the relation and differences would clarify novelty.
  • Router post-training and representation-injection PEFTs (e.g., RoMA, Echo-LoRA), and task/weight vector approaches (e.g., ChatVector) are highly related and should be acknowledged and, where feasible, compared.
Questions for authors
  • How sensitive is expert selection to the λ weights in Eq. (2), the activity threshold τ, and coverage threshold γ? Could you include a sensitivity analysis or report default values used?
  • Did you try learning the projection matrices Pℓ (e.g., small linear maps or bottlenecks shared across tasks) instead of fixing them via random semi-orthogonal matrices? If so, how did this affect performance and trainable parameter counts?
  • How were the specific source layers chosen for extraction? Would learned or data-driven layer alignment between source and target yield further gains over uniform spacing?
  • Can you report inference-time latency and memory overhead relative to the target baseline (with and without injected experts), and training wall-time vs PEFT baselines?
  • How would LLM2llm operate when the source is a dense LLM without explicit experts? Could “pseudo-experts” (e.g., clustered MLP channels) be extracted to extend applicability?
  • Could you compare against training-free or light-training merging alternatives (e.g., Sub-MoE-like fusions, task/weight vectors) and router post-training (e.g., RoMA) to position LLM2llm among non-distillation transfer methods?
  • Are the condensed experts extracted once and reused unchanged across all tasks in Tables 1–2, or were they re-extracted per task/domain? Please clarify the reuse protocol and any task-specific tuning.
  • How robust is performance when domain calibration corpora differ significantly from downstream tasks (e.g., removing specialized domains like legal/medical during calibration)?
6 comments
AuthorsOpenReview ↗

Response to Reviewer c6Tx [1/3]

Response to Reviewer #c6Tx

Thank you for your careful reading and insightful questions. Thank you for the time and care you devoted to reviewing our paper. Your constructive feedback helped us strengthen our evaluation and clarify what our results support.

Q1 Expert selection and calibration. (Question 1 & Question 8 & Weakness 7)

R1 (a) Expert selection hyperparameters. In Eq. (2), we use weights of ​0.4 for coverage, 0.4 for lower-quartile routing activity, and 0.2 for routing entropy​. The activity threshold is (τ=1.0)(\tau=1.0) on the standardized routing activity, and the minimum coverage is ​(γ=0.5)(\gamma=0.5). With five calibration domains, an expert must exceed the activity threshold in at least three domains to qualify for selection. The score then ranks eligible experts by the breadth, consistency, and balance of their routing activity.

(b) Calibration domains and downstream transfer. We sampled 20,000 tokens per domain from GitHub, arXiv, Wikipedia, C4, and B3G, for a total of ​100,000 tokens​. We used this same calibration pool to extract the experts used for the legal and medical tasks. With OLMoE-7B as the target, LLM2llm achieved 84.98% on CaseHold and ​48.63% on MedMCQA​. These results show that the extracted experts can support downstream tasks in these domains using a common calibration pool. Sensitivity to changing the composition of that pool remains an open empirical question. We will clarify the calibration setup and correct the per-domain token count in Appendix A.

Q2 Fixed and learnable projections. (Question 2 & Weakness 2)

R2 Yes. On OLMoE-7B, we compared the fixed projections with ​trainable per-layer projections initialized from the same QR matrices​. Both variants fine-tuned the transferred experts on the same SFT data for the same number of optimization steps, with the target backbone frozen.

ProjectionTrainable parametersWinoGrande Acc. (%)PIQA Acc. (%)
Fixed QR37.8M64.0186.45
Trainable projection, same QR initialization62.97M60.3784.72

Learning the projections added 25.17M trainable parameters and reduced accuracy by 3.64 percentage points on WinoGrande and ​1.73 points on PIQA​. One possible explanation is that a fixed mapping lets the experts adapt through a stable interface, whereas joint training requires the experts and projections to adapt together. The fixed projection performed better under the tested SFT budget.

Q3 Source-layer selection and expert alignment. (Question 3 & Weakness 3 & Weakness 7)

R3

R3(a). Source-layer selection and source–target mapping. As described in Appendix B.2, we select four uniformly spaced layers across the depth of the source MoE and condense the selected experts in each layer into one transferable expert. We inject these experts into approximately uniformly spaced target layers, ​preserving their depth order​. For OLMoE-7B, we use layers 1, 5, 9, and 13 with 1-based indexing.

Our rationale is that ​experts at shallow, middle, and deep layers can encode complementary information​. Sampling across the source model’s depth gives us access to experts from multiple stages of representation processing. Distributing the injected experts across the target model’s depth allows them to contribute to downstream adaptation at multiple stages during SFT.

Table 4 holds the four source experts fixed and varies the target injection locations. ​Uniform injection achieved the highest average accuracy of 81.99%, compared with 79.38% for early layers, 63.42% for late layers, and 80.42% for random layers. These results support distributing the experts across the target model’s depth. Whether a learned or activation-based layer mapping can improve on this placement remains an empirical question.

R3(b). Channel alignment within a source layer. We align intermediate channels across experts from the same source layer before condensation. Removing this alignment lowered accuracy by 10.63 percentage points on WinoGrande and ​0.92 points on PIQA​. When selected experts have weak channel correspondence or little shared weight structure, the benefits of channel alignment and low-rank condensation may be smaller.

AuthorsOpenReview ↗

Response to Reviewer c6Tx [2/3]

(continued from [1/3])

Q4 Deployment costs and expert reuse. (Question 4 & Question 7 & Weakness 6)

R4

R4(a). Inference latency and memory. Thank you for asking about the cost of keeping the injected experts active at inference. We benchmarked standard OLMoE-7B and LLM2llm on a single NVIDIA H20-3e in BF16. The LLM2llm variant used the ​same four injection layers, expert dimensions, and fixed projection design as our main experiments**​, with randomly initialized expert weights for timing. We measured 30 paired requests with identical prompt and continuation tokens, batch size 1, a 1,024-token prompt, a 100-token continuation, and KV caching.

VariantTTFT (ms, P50)Decode latency (ms/token, P50)100-token request latency (s, P50)Peak allocated GPU memory (GiB)
Standard OLMoE-7B39.4314.3591.461113.186
LLM2llm (four expert branches)40.0614.8621.511413.303
Relative overhead+1.60%+3.50%+3.45%+0.89%

LLM2llm offers a favorable trade-off between downstream accuracy and deployment cost. With all four expert branches active, median latency for a 100-token request increased by only ​**3.45%**​, and peak GPU memory increased by only 120 MiB (0.89%) in our benchmark. Expert extraction is a one-time cost that can be shared across downstream tasks. At inference, deployment requires only OLMoE-7B and the four compact expert branches.

R4(b). Expert preparation and reuse. Expert preparation takes approximately ​130 minutes​, including 40 minutes for routing calibration and 90 minutes for extraction and condensation. This cost is incurred once per source checkpoint and extraction configuration. Downstream tasks reuse the resulting expert weights as initialization and fine-tune their own copies.

Q5 Source-model requirements. (Question 5 & Weakness 1)

R5 ​MoE is a common architecture among today’s leading large-scale open-weight LLMs​, including Qwen3.5-122B-A10B and DeepSeek-V3. Sparse routing lets these models draw on a large parameter capacity while activating only a fraction of their parameters per token. For LLM2llm, ​the explicit experts and routing decisions also make pretrained MoEs natural sources for studying how expert usage varies across tasks and identifying generalist experts that are consistently active across domains​.

A dense model can also serve as a source after conversion to a routed MoE. MoEfication partitions dense FFN channels into functional groups and builds routers to select among them. LLaMA-MoE partitions the FFNs of LLaMA-2 7B into experts, adds routers, and continues pretraining. Once the converted model provides accessible expert weights and token-level routing decisions, LLM2llm can use those signals to score the resulting pseudo-experts across calibration domains and condense the selected experts for transfer.

AuthorsOpenReview ↗

Response to Reviewer c6Tx [3/3]

(continued from [2/3])

Q6 Baseline fairness. (Weakness 4)

R6 Thank you for raising these concerns about the training schedule, LoRA ranks, and backbone comparability.

(a) Full fine-tuning and checkpoint selection. The three-epoch SFT schedule applies to the language-understanding experiments. The original DollyEval runs used ten epochs. To test whether the shared schedule disadvantaged Full FT on DollyEval, we compared two learning rates and selected the best checkpoint for each by ​validation ROUGE-L​. We applied the same checkpoint selection criterion to LLM2llm.

MethodLearning rateCheckpointValidation ROUGE-LDollyEval ROUGE-L
Full FT, original setting2e-5Epoch 10, final22.8022.58
Full FT2e-5Epoch 6, best on validation23.2022.65
Full FT1e-5Epoch 8, best on validation22.6522.10
LLM2llm2e-5Epoch 10, best on validation31.8031.75

The original learning rate of 2e-5 remained the better of the two tested rates. Selecting epoch 6 by validation ROUGE-L improved Full FT by 0.07 points on DollyEval, while LLM2llm retained a 9.10-point lead over the best Full FT setting tested. Given the small gain from earlier checkpoint selection, we will soften the attribution to overfitting in our discussion of Figure 3.

(b) LoRA ranks and trainable parameter budgets. Table 1 evaluates two LoRA ranks per backbone. LoRA-r8 on OLMoE-7B updates 1.10% of the target parameters and achieves 65.82% average accuracy, compared with 73.95% for LLM2llm at ​0.54%. LoRA-r16 on LLaMA2-7B updates 0.25% and achieves 42.30% average accuracy, compared with 61.12% for LLM2llm at ​0.19%. LLM2llm thus outperforms the higher-budget LoRA configuration tested on each backbone.

(c) ESFT and backbone comparability. On OLMoE-7B, ESFT and LLM2llm use the same pretrained MoE backbone. On LLaMA2-7B, the ESFT result uses an MoE-converted variant because ESFT adapts routed experts. We will clarify this distinction in Table 1 and the accompanying discussion. Our same-backbone comparisons on dense LLaMA2-7B use Full FT, LoRA, and PiSSA.

Q7 Comparison with non-distillation transfer methods. (Question 6 & Weakness 5 & Weakness 8 & Weakness 9)

R7

R7(a). Controlled comparison with training-free expert fusion. Thank you for suggesting these comparisons. Table 3 includes ​Direct Fusion​, a training-free fusion baseline evaluated under the same downstream adaptation setup. It uses the same selected source experts as LLM2llm, aligns their channels, and uniformly averages the aligned weights. Both variants use the same residual branches, fixed projections, and injection coefficient β. On Qwen3.5-122B → OLMoE-7B, ​LLM2llm improves average accuracy from 72.38% to 75.30% across five tasks​. This comparison measures the combined benefit of joint SVD and score-weighted fusion over aligned averaging.

R7(b). Relation to expert compression, weight arithmetic, and router adaptation.

  • Expert compression. Sub-MoE uses joint SVD to merge groups of similar experts, while MoE-I² prunes experts and applies low-rank decomposition to the remaining ones. Both aim to compress an existing MoE. LLM2llm uses joint SVD to construct ​reusable expert initializations for a separate pretrained target​. Sub-MoE · MoE-I²
  • Weight arithmetic and representation injection. Chat Vector transfers a weight difference derived from a base model and its corresponding chat model. Echo-LoRA injects deeper-layer activations into shallower PEFT modules within the same model. LLM2llm ​transfers pretrained expert weights as residual branches​, using projections to bridge differences in hidden size. Chat Vector · Echo-LoRA
  • Router post-training. RoMA updates the routers of a pretrained MoE while freezing its existing experts. LLM2llm freezes the target backbone and ​fine-tunes only the injected source-derived experts​, which allows it to adapt both dense and MoE targets. RoMA

We will clarify these relationships in the revision. A direct comparison with Sub-MoE-style fusion would use the same source–target pair, residual branch budget, and SFT protocol as LLM2llm.

Thank you again for your thoughtful review. We wish you all the best in your research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.