- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose LLM2LLM, a parameter-efficient adaptation framework that aims to transfer knowledge from a larger Mixture-of-Experts (MoE) model to a smaller pre-trained model, without relying on teacher-generated supervised learning. The core hypothesis is that experts consistently activated across multiple domains contain generalizable knowledge that can be reused by other models.
LLM2LLM consists of three stages. First, it identify domain-generalizable experts using routing statistics based on coverage, robustness, and entropy. Second, it combine the selected experts through channel alignment, SVD-based shared subspace extraction, and score-weighted fusion. Third, to accommodate differing hidden dimensions, it use fixed random projections to inject the resulting experts into the pre-trained target model as residual branches. During the subsequent fine-tuning process, the original target model remains frozen, and only the injected experts are updated.
The authors evaluated LLM2LLM using two source-target pairs: Qwen3.5-122B to OLMoE-7B, and Qwen3.5-35B to LLaMA2-7B. LLM2LLM achieved higher average scores on benchmarks for language understanding and conversational generation compared to Full Fine-Tuning, LoRA, PiSSA, and ESFT, while updating only 0.54% and 0.19% of the target model's parameters, respectively. Ablation studies confirm that source-based initialization, expert selection, and condensation all contribute to the observed performance improvements. Overall, these experiments demonstrate that initializing additional adaptation modules using selected and integrated source-expert parameters can enhance subsequent performance. Whether this improvement reflects the identification and transfer of generalizable knowledge, or simply the benefits of using pre-trained parameter initializations, remains an important distinction.
Critical strengths
- Multi-criteria approach for selecting reusable experts.
The strategy for selecting experts is persuasive. Rather than relying solely on average routing frequency, the authors evaluate cross-domain generality from three complementary perspectives. First, 'coverage; measures the breadth of an expert’s activation across different domains. Second, 'robustness' captures whether an expert remains active even in less favorable domains. Third, 'entropy' assesses how evenly distributed an expert’s routing activity is. These criteria provide a valid method for distinguishing between experts that are activated broadly and consistently, and those confined to only a few domains. Using the selected experts to construct source-derived initialization weights is a practical and well-motivated design choice.
- Promising empirical results for source-based expert initialization and parameter-efficient adaptation.
This method outperforms reported fine-tuning baselines while updating only 0.19% to 0.54% of the target parameters. Moreover, an ablation study reveals that replacing the transferred expert modules with randomly initialized modules of the same architecture and size causes the average accuracy to drop from 75.30 to 52.46. This significant performance gap demonstrates that pre-trained source expert parameters provide a meaningful initialization that goes beyond merely increasing learning capacity.
Critical weaknesses
The central idea of identifying broadly activated experts from a pre-trained MoE, consolidating their parameters, and transferring them to another model has already been explored by XPERT. Furthermore, prior expert-merging methods such as M-SMoE have investigated routing-guided expert selection, channel alignment, weighted merging, and low-rank compression. While LLM2LLM introduces different selection criteria and condensation procedures, the paper does not sufficiently explain what is fundamentally new about these components. Its clearest distinction appears to be injecting source-derived experts into an already pre-trained target model for parameter-efficient adaptation. However, the paper does not compare its extraction procedure against XPERT-based alternatives under the same adaptation framework. This makes it difficult to determine whether the proposed selection and condensation methods constitute substantial methodological contributions beyond existing techniques.
The paper assumes that experts consistently activated across domains contain generalizable knowledge and that dominant SVD directions capture shared knowledge while suppressing expert-specific variation. Although the proposed selection criteria are reasonable, neither assumption is directly validated. More importantly, the ablation results suggest that much of the improvement may come from pre-trained expert initialization rather than the proposed knowledge selection and condensation mechanisms. Random expert selection already achieves 71.30 average accuracy compared with 75.30 for LLM2LLM, and the gains from both selection and condensation are heavily concentrated on WinoGrande. Without stronger controls, such as repeated random selections, direct analysis of shared knowledge representations, and evaluation before downstream fine-tuning, it remains unclear whether the method genuinely transfers generalizable knowledge or primarily provides a favorable initialization for additional trainable FFN modules. This distinction is central to the paper's main scientific claim.
Questions for authors
-
Section 3.1 assumes that experts consistently routed across domains contain generalizable knowledge. However, 'Probing Semantic Routing in Large Mixture-of-Expert Models' shows that routing depends on semantic similarity, while 'Decoding Knowledge Attribution in Mixture-of-Experts' suggests that different knowledge relations may activate distinct neurons within the same expert. Have the authors measured whether their selected generalist experts actually use overlapping neurons or activation subspaces across domains? Could their high routing scores instead reflect shared semantic features or domain-specific computations within the same expert?
-
In Table 3, Random Selection already achieves 71.30 accuracy compared with 75.30 for LLM2LLM. Moreover, excluding WinoGrande reduces the average improvement from 4.00 to 1.80 points. How stable are these differences across multiple random expert selections and training seeds? Does the proposed generalist score correlate with downstream transfer performance across experts?
-
Appendix C assumes that recurring parameter structures dominate the leading SVD directions, while expert-specific variations contribute weaker or unstructured components. Have the authors measured whether the retained directions are actually shared across domains? Can they compare the functional contributions of retained and discarded directions to determine whether SVD preserves generalizable knowledge rather than simply high-energy parameters?
-
In Table 3, Direct Fusion achieves 72.38 compared with 75.30 for LLM2LLM, but the improvement is concentrated on WinoGrande (+10.81 points), with an average gain of only 0.95 points on the other four benchmarks. Can the authors separately ablate channel alignment, score weighting, and SVD projection to identify which component produces the improvement?
-
Random Initialization performs substantially worse than source-derived initialization, but this does not distinguish knowledge transfer from favorable pretrained initialization. Can the authors compare against an intact pretrained source expert or a pretrained target FFN injected using the same residual architecture? Additionally, what performance improvement is observed immediately after injection, before downstream SFT?
-
Equation (6) uses a fixed projection generated by QR decomposition of a random Gaussian matrix. How much does performance vary across different projection seeds? Since this projection matches dimensions without explicitly aligning semantic representations, how does its performance compare with a learned or activation-based projection under the same training budget?
-
Appendix A reports approximately 50K calibration tokens per domain from GitHub, arXiv, Wikipedia, C4, and B3G. What are the exact domain definitions, and what values are used for the selection thresholds and weights in Equation (2)? How consistent are the selected expert sets when the calibration data size or domain composition changes?
-
The experiments use two source-target pairs, with both targets being approximately 7B models. Can the same condensed experts from one source be evaluated on multiple target architectures or sizes, particularly smaller 1B-3B models? This would help establish whether the extracted components are genuinely reusable across models rather than effective only for the tested pairs.
-
Figure 3 attributes the weaker Full FT results to overfitting, despite its substantially lower training loss. Were learning rates, regularization, and checkpoint selection tuned independently for each baseline? Can the authors report validation curves and multi-seed results to distinguish overfitting from suboptimal optimization settings?
-
For generation tasks, can the authors supplement ROUGE-L with factual accuracy or instruction-following evaluations to assess whether the improvements reflect meaningful capability gains?
6 comments
Response to Reviewer PMuL [2/3]
(continued from [1/3])
Q3. Shared-subspace condensation and component ablations. (Questions 3 and 4 & Weakness 2)
R3
R3(a). What the retained directions represent (Question 3). Experts selected for broad cross-domain activity can still retain structure specific to individual experts, domains, or tasks. Selection identifies candidate experts, and condensation further refines what we transfer from them. After channel alignment, joint SVD estimates a shared low-rank basis that minimizes reconstruction error across the selected experts’ weight matrices. We project the score-weighted average onto this basis to retain dominant directions estimated jointly across experts and suppress components outside the retained subspace.
Appendix C motivates the hypothesis that recurring parameter structure is concentrated in these leading directions. Our analysis characterizes these directions in weight space. Whether they are used across domains remains an empirical question.
R3(b). Downstream effects of the design choices (Question 4 & Weakness 2). Direct Fusion starts from the same score-selected experts, retains channel alignment, and uses the same injection coefficient (\beta) as LLM2llm. It then averages the aligned weights uniformly in the original parameter space.
| Variant | Channel alignment | Expert selection | SVD projection | WinoGrande Acc. (%) | PIQA Acc. (%) |
|---|---|---|---|---|---|
| LLM2llm | Yes | Generalist score | Yes | 64.01 | 86.45 |
| Without channel alignment | No | Generalist score | Yes | 53.38 | 85.53 |
| Random Selection | Yes | Random | Yes | 51.22 | 84.11 |
| Direct Fusion (uniform averaging) | Yes | Generalist score | No | 53.20 | 85.20 |
Channel alignment puts intermediate channels into correspondence before fusion. Removing it lowers accuracy by 10.63 points on WinoGrande and 0.92 on PIQA. Generalist-score selection favors experts with broad and consistent routing activity. Replacing it with random selection lowers accuracy by 12.79 and 2.34 points, respectively.
Shared-subspace condensation improves on Direct Fusion across all five tasks in Table 3, including +1.25 points on PIQA and **+10.81 points on WinoGrande. Starting from the same selected and aligned experts, these gains support the value of refining their aggregation before transfer. The comparison captures the combined contribution of SVD projection and score-weighted fusion**.
Q4 Pretrained initialization and performance before SFT. (Question 5 & Weakness 2)
R4
Thank you for suggesting a comparison with pretrained initializations. We use the same residual branch architecture, injection layers, trainable parameter count, and SFT budget across variants. For Intact Source Expert, we directly copy the highest-scoring Qwen expert from the selected set at each injection layer. For Target-native Expert Init, we initialize each branch with the most frequently routed expert from the corresponding OLMoE layer on the calibration data. We match the initial residual output RMS of the pretrained variants to that of LLM2llm on the same calibration samples.
| Branch initialization | WinoGrande Acc. (%) after SFT | PIQA Acc. (%) after SFT |
|---|---|---|
| Random Init | 51.14 | 49.46 |
| Intact Source Expert | 57.53 | 85.09 |
| Target-native Expert Init | 49.57 | 79.30 |
| LLM2llm | 64.01 | 86.45 |
LLM2llm outperforms Intact Source Expert by 6.48 and 1.36 percentage pointson WinoGrande and PIQA, respectively. Both variants use experts from the same Qwen checkpoint, suggesting that condensing multiple selected experts provides a more effective initialization than transferring the highest-scoring expert alone. One possible explanation is that the selected experts contribute complementary information across domains. LLM2llm also improves over Target-native Expert Init by 14.44 and 7.15 percentage points, showing that the full transfer pipeline offers a benefit over initializing the same branch with the target model’s own pretrained expert.
Does expert injection improve accuracy before any downstream SFT?
Under our strict answer-matching protocol, accuracy was near zero both for the base models and immediately after expert injection. Since this metric also depends on whether the model follows the requested answer format, the pre-SFT scores provide limited evidence about which capabilities the injected experts carry. The measurable benefit appears during downstream adaptation. Under the same SFT budget, the transferred experts provide a stronger initialization that leads to higher final task accuracy. Our results therefore support the use of transferred experts for supervised adaptation, while immediate transfer of behaviors acquired during source-model post-training, such as instruction following, remains unestablished.
Response to Reviewer PMuL [3/3]
(continued from [2/3])
Q5 Sensitivity to projection seeds and comparison with a learned projection. (Question 6)
R5 The fixed QR projection in Eq. (6) maps between the source and target hidden dimensions. During SFT, the transferred expert weights adapt to the target through this fixed mapping. To measure sensitivity to the projection, we evaluate three QR seeds while keeping the condensed experts, SFT seed, and training configuration fixed. For the learned-projection comparison, we use one of these frozen runs as the reference and initialize the trainable projection with the exact same QR matrix. Both variants update the expert weights using the same SFT data and number of optimization steps.
| Projection | Trainable parameters | WinoGrande Acc. (%) | PIQA Acc. (%) |
|---|---|---|---|
| Fixed QR, 3 seeds (mean ± SD) | 37.8M | 64.14 ± 0.11 | 86.39 ± 0.18 |
| Fixed QR, paired reference | 37.8M | 64.01 | 86.45 |
| Trainable projection, same QR initialization | 62.97M | 60.37 | 84.72 |
The fixed projection shows low sensitivity to the three tested seeds, with standard deviations of 0.11 and 0.18 percentage points on WinoGrande and PIQA, respectively. In the paired comparison, training the projection reduced accuracy by 3.64 points on WinoGrande and 1.73 points on PIQA, while adding 25.17M trainable parameters.
One possible explanation is that freezing the projection gives the pretrained experts a stable mapping through which to adapt to the target. When the projection is trainable, SFT must also learn that mapping from a random QR initialization, jointly with adapting the expert weights. Under the same data and step budget, this joint optimization may be harder than adapting the expert weights through a fixed mapping. The paired result favors the frozen projection under the tested SFT budget.
Q6. Reusing the same condensed experts across target models. (Question 8)
R6 Our experiments already cover both a MoE target (OLMoE-7B) and a dense target (LLaMA2-7B). In both cases, the Qwen source and the target belong to different model families. The condensed experts are injected as residual branches, and a target-specific projection handles differences in hidden width. This allows the source and target to differ in size, model family, and architecture.
Expert extraction and condensation are performed once per source checkpoint. The resulting expert weights can serve as a shared initialization for different tasks on the same target and for targets of different sizes or architectures. For each new target, we configure the injection layers and projection, then fine-tune a separate copy of the same condensed weights.
Our two experiments demonstrate transfer to both dense and MoE targets using separately extracted expert sets. Reusing the exact same condensed weights on a 1B–3B target would directly test how well this reuse works at a smaller scale.
Q7. Full fine-tuning and model selection (Question 9)
R7. In the original OLMoE-7B experiments on Dolly, Full FT and LLM2llm used a shared learning rate of 2e-5 and were evaluated at epoch 10. In the follow-up, we tested 1e-5 and 2e-5 for Full FT at fixed weight decay. We selected the Full FT learning rate and both methods’ checkpoints by ROUGE-L on a held-out Dolly validation split, then evaluated the selected checkpoints on DollyEval.
| Method | Learning rate | Checkpoint | Validation ROUGE-L | DollyEval ROUGE-L |
|---|---|---|---|---|
| Full FT | 2e-5 | Epoch 10 → Epoch 6 (best on validation) | 22.80 → 23.20 | 22.58 → 22.65 |
| Full FT | 1e-5 | Epoch 8 (best on validation) | 22.65 | 22.10 |
| LLM2llm | 2e-5 | Epoch 10 (best on validation) | 31.80 | 31.75 |
Validation favored the original 2e-5 learning rate for Full FT. Selecting its epoch-6 checkpoint improved DollyEval ROUGE-L by 0.07 points. With both methods using validation-selected checkpoints, LLM2llm retained a 9.10-point advantage.
Figure 3 shows that Full FT’s training loss continued to fall while DollyEval ROUGE-L largely plateaued. The follow-up shows a modest late-epoch decline in validation ROUGE-L, with little change on DollyEval when selecting the earlier checkpoint. We will present overfitting as a possible explanation for the late-training behavior.
Q8 What do the generation results measure? (Question 10)
R8 Generation evaluation. Our generation experiments report ROUGE-L, which measures overlap with the reference responses. The observed gains therefore indicate closer agreement with those references. We will clarify this scope in the revision. Factual accuracy and instruction following would require separate evaluations.
Thank you again for your thoughtful review. We wish you all the best in your research.
Response to Reviewer PMuL [1/3]
Response to Reviewer #PMuL
Thank you for your careful reading and insightful questions. Thank you for the time and care you devoted to reviewing our paper. Your constructive feedback helped us strengthen our evaluation and clarify what our results support.
Q1 Novelty and relation to prior work. (Weakness 1)
R1 We thank the reviewer for asking us to clarify how LLM2llm relates to XPERT and M-SMoE. Our central contribution is a method for adapting an existing pretrained model by transferring expert weights from a separate source model into trainable residual branches.
Q2. Cross-domain expert selection and downstream transfer. (Questions 1, 2, and 7 & Weakness 2)
R2
R2(a). Generalist scoring and downstream transfer (Questions 1 and 2 & Weakness 2). The generalist score measures how broadly and consistently an expert is routed across calibration domains. It combines domain coverage, lower-quartile routing activity, and routing entropy. We use these signals to identify candidates for cross-domain transfer.
In Table 3, Generalist Selection achieves 75.30% average accuracy across five tasks, compared with 71.30% for Random Selection. Excluding WinoGrande, the average gain is 1.80 percentage points. To assess sensitivity to the choice of random experts, we sampled three expert sets with different seeds, holding the number of experts per layer, condensation procedure, projection, and SFT seed fixed.
Generalist Selection outperformed the mean of the three random draws by 12.58 percentage points on WinoGrande and 2.50 points on PIQA. These results support the downstream benefit of the generalist selection rule at the level of the selected expert sets.
R2(b). Calibration domains and sample size (Question 7). We drew on OLMo’s open pretraining corpus when choosing GitHub, arXiv, Wikipedia, C4, and B3G as calibration sources. These sources cover different types of text and subject matter. We sampled 20,000 tokens per domain, for a total of 100,000 tokens, to reduce the influence of individual documents on the routing statistics. We will correct the per-domain token count in Appendix A.