acceptodds
Under review as a conference paper at ICLR 2027

Transfer: Coefficient Transfer for Efficient Model Merging

Abstract

Model merging offers a promising solution for combining multiple fine-tuned checkpoints into a single model through parameter arithmetic. However, finding optimal merging coefficients requires an extensive search that becomes prohibitively expensive as model size and number increase, due to high memory requirements and combinatorial growth in the search space. We show that, within the same model family, models exhibit highly similar performance distributions over merging coefficients across different model sizes. This distributional similarity enables a practical paradigm we call \textit{\alphaTransfer}: search for optimal coefficients on a small proxy model, then directly transfer them to larger target models in the same family. We verify Transfer across multiple merging methods, model families, and tasks. Experimental results demonstrate a 6 speedup and 70% memory reduction on vision transformers, and a 20 speedup and 85% memory reduction on large language models, while maintaining comparable performance. Our findings establish Transfer as an efficient and generalizable approach to scaling model merging.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qrPxOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem of finding good merging coefficients, i.e., how to weigh different task vectors, in model merging. Specifically, the paper proposes to search for optimal merging coefficients using a smaller model variant from the same model family and then transferring the coefficients to a larger model. In this way, expensive search on the larger model can be avoided, which allows either less compute requirements overall, or more evaluations, i.e., potentially better coefficients, with the same compute budget. To account for different model depths, the paper proposes to match coefficients at the same relative depth and interpolate them. Experiments on image-test encoders and large language models confirm the method works, i.e., the coefficients transfer, and actual compute savings can be achieved.

Critical strengths

Generally, I find the paper very clear and to-the-point:

(S1) The problem and the solution are intuitive and well-motivated, but not trivial, the experiments are sufficient to support the claims, and the paper also considers potential obstacles, like different model depth. The supplementary material reports the hyperparameters and complete results in detail, which I found helpful. While reading, the ablation in Sec 4.5 (using the transferred coefficients as initialization for coefficient search on the larger model) came to my mind early on, so it was nice this was actually included.

(S2) The paper addresses a real and practically impactful problem by a natural solution that can be easily implemented. Here, I like that the solution does not introduce architectural complexity or complicated methods. Therefore, I expect this finding will be helpful in practice, so communicating it via ICLR is useful and beneficial to the community.

Critical weaknesses

While I find this paper clearly meets the bar for acceptance, my largest concern is about transfer to other merging methods beyond task vector averaging. For example, recent developments include subspace projections in weight [1, 2] and activation space [3], where different hyperparameters apply. Task Arithmetic and variants are common, and they are simple and practical, so it's reasonable to develop methods tailored to these, but it remain open how future-proof the paper is. This is relevant, because it may reduce the impact of the paper.


References:
[1] Gargiulo et al. "Task singular vectors: Reducing task interference in model merging." CVPR 2025
[2] Skorobogat et al. "Subspace-boosted model merging." arXiv 2025
[3] Wei et al. "SSR-Merge: Subspace Signal Routing for Training-Free LoRA Merging in Diffusion Models." ICML 2026

Questions for authors

(Q1) Do the authors think, or have evidence, that hyperparameter transfer also works for other merging parameters than task vector coefficients, e.g., for the subspace methods cited above? I would be very curious about a discussion of this aspect.

(Q2) While reading, I was also very curious about the limitations. In particular, the paper already mentions

may alter the optimization landscape, making direct coefficient mapping less effective

I would be curious what are the factors that lead to this. For example, we know that LLMs of the same family behave similarly, because they are trained on similar data with similar settings. However, how far can models be apart to still support coefficient transfer? The paper also mentions "cross architectures" (line 134), which probably refers to models with different depth, but I would be curious if this also works for models that have slightly different components, such as linear attention instead of standard attention. In summary, I'm interested in whether (especially task-wise) coefficients depend more on the (task) data or the model.

Perhaps the authors could compare task-wise coefficients across architectures?

1 comment

Weakness 1

We thank the reviewer for the positive assessment and for pointing us to these recent subspace-based methods. To evaluate whether α\alphaTransfer extends beyond task-vector averaging, we applied it to TSV-M and to Subspace Boosting on top of Task Arithmetic and TIES-Merging:

Proxy → TargetMethodAccuracy (%) (Origin → Transfer)Time (h) (Origin → Transfer)SpeedupMemory (GB) (Origin → Transfer)Reduction
ViT-B/32 → ViT-L/14TSV-M86.29 → 85.746.96 → 1.285.46×2.12 → 0.6668.7%
Subspace Boosting (TA)85.39 → 85.396.89 → 1.295.33×2.12 → 0.6668.7%
Subspace Boosting (TIES)81.85 → 79.286.94 → 1.255.55×2.12 → 0.6668.7%
ViT-B/16 → ViT-L/14TSV-M86.29 → 84.946.96 → 1.853.77×2.12 → 0.8758.9%
Subspace Boosting (TA)85.39 → 85.396.89 → 1.913.61×2.12 → 0.8758.9%
Subspace Boosting (TIES)81.85 → 79.286.94 → 1.903.65×2.12 → 0.8758.9%
SigLIP-Base → SigLIP-LargeTSV-M86.69 → 86.696.14 → 2.332.63×2.84 → 1.0762.2%
Subspace Boosting (TA)84.87 → 84.876.22 → 2.332.67×2.84 → 1.0762.2%
Subspace Boosting (TIES)80.02 → 80.026.29 → 2.322.71×2.84 → 1.0762.2%

Across all settings, α\alphaTransfer achieves a 2.6$\times$ to 5.6 ×\times speedup and a 59% to 69% memory reduction. The accuracy is preserved within 1.35 points in all cases except Subspace Boosting (TIES) on ViT, where the drop is 2.57 points.

In contrast, activation-space methods such as SSR-Merge compute their merging components in closed form from activation statistics, requiring only a single pass over calibration data. As there is no costly search to accelerate, they fall outside the scope of α\alphaTransfer, though whether such statistics share structure across scales is an interesting direction for future work.

Question 1

We thank the reviewer for this interesting question. While we have not yet systematically studied this, we believe that transfer is likely to extend beyond merging coefficients. Some hyperparameters are defined in relative terms (e.g., the retained rank ratio in TSV-M) and are thus comparable across model sizes. Moreover, since models within the same family are fine-tuned on the same data, we hypothesize that the relative ranking of such hyperparameters is largely preserved across scales, as we observe for merging coefficients. We leave a systematic study of this direction to future work.

Question 2

We thank the reviewer for the thoughtful questions.

Clarification on “cross architectures.”

We use the phrase “cross architectures” in line 134 to express that coefficient transfer within a model family holds consistently across different architectures (ViT, Qwen3), rather than referring to models with different depth. We will revise the wording to avoid this confusion.

Dependence of coefficients on data or model

To investigate this question, we compared the Spearman correlation of coefficient rankings across model families:

GroupModel PairSpearman ρ\rho
Same familyViT-B/16 – ViT-L/140.84
Same familyViT-B/32 – ViT-L/140.58
Same familySigLIP-Base – SigLIP-Large0.79
Cross familyViT-B/16 – SigLIP-Base0.83
Cross familyViT-B/32 – SigLIP-Base0.56
Cross familyViT-L/14 – SigLIP-Large0.74

The results show that cross-family correlations can also be as high as within-family correlations, suggesting that the coefficient landscape is largely shaped by the task data. This also indicates that α\alphaTransfer may be applicable even beyond a single model family, which we consider a promising direction for future work.

Regarding the differences in individual components such as linear attention, a controlled ablation would require pre-training models that differ only in that component under an identical setup. Since our experiments rely on publicly available pre-trained models, such an ablation is beyond our current scope.

Comparing task-wise coefficients

We note that directly comparing the optimal coefficients across models may not be informative, as the optimum found depends on the search algorithm. For example, on a non-convex landscape, optimization-based methods such as AdaMerging may converge to quite different points across models, even when the overall landscapes are similar. Indeed, we observe that the coefficients found on different models are often not close to each other, yet transfer remains effective. This suggests that what transfers is the overall shape of the landscape, under which coefficients found on the small model also perform well on the large model, rather than the exact location of the optimum.

Reviewer_cd6NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes αTransfer, which searches or optimizes merging coefficients on smaller proxy models and applies those coefficients to larger models from the same family. The underlying merging methods remain unchanged. Global and task-wise coefficients transfer directly, while layer-wise coefficients are mapped through relative-depth copying or interpolation. The paper also explores target-side refinement initialized from transferred coefficients. Experiments cover CLIP-ViT, SigLIP, and Qwen3 across eight vision tasks and four language tasks. The main contribution is empirical evidence that coefficient transfer can reduce search time and memory while preserving merging performance in several settings, although the accuracy trade-off varies across methods and proxy-target pairs.

Critical strengths
  1. The paper identifies a useful way to reduce the cost of coefficient selection without redesigning the underlying merging algorithm. Its application to both evaluation-based and gradient-based methods, together with support for global, task-wise, and layer-wise coefficients, makes the approach relevant to several existing merging procedures. The empirical observation that coefficients can transfer across model sizes is the main contribution and could be practically valuable when suitable proxy experts are available.

  2. Several experiments demonstrate a favorable accuracy-cost trade-off. For SigLIP-Base to SigLIP-Large with TIES, Table 3 reports 77.39% accuracy after transfer compared with 77.50% for direct target search, with a 3.01× search speedup. For Qwen3-0.6B to Qwen3-4B with Task Arithmetic, Table 4 reports identical average accuracy of 76.03% and an 11.03× search speedup. The layer-wise and hybrid experiments also provide useful evidence that the approach can extend beyond direct transfer of a single scalar.

Critical weaknesses
  1. The language-model results contain unresolved numerical inconsistencies that affect a central empirical claim. For Qwen3-0.6B to Qwen3-4B with TIES-Pairwise, Tables 4 and 8 report 73.69% for both direct target search and coefficient transfer. However, Table 10 reports corresponding averages of 60.42% and 60.29%. For standard TIES on the same model pair, the transfer accuracy is 65.26% in Table 4 but 64.50% in Table 10. These may be reporting errors or differences in experimental configuration, but the manuscript does not explain them. Consequently, it is unclear which results support the claimed preservation of language-model performance at the reported search cost.

  2. The conditions under which coefficient transfer remains reliable are insufficiently characterized relative to the paper’s broad interpretation of the finding. Table 2 measures rank correlation across coefficient configurations, which does not establish that the proxy-selected configuration is near-optimal on the target. Table 3 contains substantial losses, including a drop from 79.10% to 72.08% for AdaMerging++ on ViT-B/32 to ViT-L/14. The sparsity ablation, layer-wise transfer, and hybrid search provide useful mitigation evidence, but they do not establish how transfer depends on the choice and training of the proxy experts. Appendix A reports independently selected fine-tuning configurations, yet there is no controlled analysis that varies the proxy training recipe while holding the target experts fixed. Because cross-scale transferability is the main contribution, this limits confidence in applying the approach beyond the evaluated checkpoint collections.

Questions for authors
  1. Can you reconcile the discrepancies between Tables 4, 8, and 10, particularly the Qwen3-4B TIES-Pairwise results? Please clarify whether these tables use the same checkpoints, merging configurations, and evaluation protocol, and provide corrected per-task and average results corresponding to the reported timing measurements.

  2. With the target experts held fixed, how sensitive is coefficient transfer to the proxy experts’ fine-tuning learning rate, training duration, and checkpoint selection? Do you have controlled results that help explain when transfer preserves performance and when losses such as those in Table 3 occur?

  3. When corresponding proxy experts must be created, what is the additional cost of training and selecting those checkpoints, and how does that change the total cost comparison with direct target search? Alternatively, please clarify whether the intended efficiency claim assumes that suitable proxy experts are already available.

5 comments
Reviewer_cd6NOpenReview ↗

Clarification about the edited copy

I have read the latest version of the manuscript and noticed that it has been edited. In particular, the Qwen3-4B per-task results in Table 10 have changed and now match the averages reported in Tables 4 and 8.

Could you confirm whether these values were corrected in response to the discrepancies raised in my review?

AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the insightful review, and apologize for the delayed response, as we ran additional controlled experiments to address the reviewer’s questions as fully as possible. We hope the new results help clarify the conditions under which transfer is reliable. Our responses are as follows.

Weakness 1 & Question 1

We thank the reviewer for pointing out this inconsistency. This is a reporting error on our part: when preparing Table 10, we mistakenly copied results obtained under a different hyperparameter setting. The correct results are those in Tables 4 and 8 (e.g., 73.69% for both direct search and transfer with TIES-Pairwise, and 65.26% for transfer with TIES on Qwen3-0.6B → Qwen3-4B), and we have corrected Table 10 in the revision accordingly. We have re-checked all LLM results, and our conclusion remains unchanged.

Weakness 2 & Question 2

To provide a controlled analysis of how the fine-tuning recipe affects transfer, we fix the target experts (ViT-L/14, fine-tuned with lr 1e-5 for 10 epochs) and vary the training recipe of the proxy experts (ViT-B/32 and ViT-B/16). For AdaMerging on checkpoints fine-tuned with lr 3e-5, we run 1,500 optimization steps instead of the 500 used in the paper, as it converges more slowly.

Training duration

We fine-tuned the proxy experts with lr 1e-5 for 10 epochs, and use the intermediate checkpoints saved at epochs 1, 3, 5, and 10.

Proxy → TargetMethodProxy EpochsAccuracy (%) (Origin → Transfer)Δ\Delta Accuracy
ViT-B/32 → ViT-L/14Task Arithmetic176.0 → 76.00.0
376.0 → 76.00.0
576.0 → 76.00.0
1076.0 → 76.00.0
AdaMerging175.7 → 73.3−2.4
375.7 → 76.5+0.8
575.7 → 76.3+0.6
1075.7 → 76.6+0.9
ViT-B/16 → ViT-L/14Task Arithmetic176.0 → 76.00.0
376.0 → 76.00.0
576.0 → 76.00.0
1076.0 → 76.00.0
AdaMerging175.7 → 70.6−5.1
375.7 → 75.5−0.2
575.7 → 75.9+0.2
1075.7 → 76.1+0.4

Transfer is robust once the proxy experts are reasonably trained. Task Arithmetic transfers without loss in all cases, and AdaMerging stays within 1 point for proxies trained for 3 epochs or more. Only under-trained proxies (epoch 1) lead to noticeable drops for AdaMerging.

Learning rate

We fine-tuned the proxy experts for 10 epochs with lr 1e-5 and 3e-5.

Proxy → TargetMethodProxy LRAccuracy (%) (Origin → Transfer)Δ\Delta Accuracy
ViT-B/32 → ViT-L/14Task Arithmetic1e-576.0 → 76.00.0
3e-576.0 → 71.1−4.9
AdaMerging1e-575.7 → 76.6+0.9
3e-575.7 → 73.2−2.5
ViT-B/16 → ViT-L/14Task Arithmetic1e-576.0 → 76.00.0
3e-576.0 → 71.1−4.9
AdaMerging1e-575.7 → 76.1+0.4
3e-575.7 → 72.7−3.0

When the proxy is fine-tuned with a larger learning rate than the target, transfer performance drops. For Task Arithmetic, the proxy selects α=0.1\alpha=0.1 instead of the target’s optimal α=0.2\alpha=0.2, resulting in a 4.9 point drop.

To understand whether this drop is caused by the learning rate itself or by the mismatch between proxy and target, we further compare a matched setting, where the target experts are fine-tuned with the same learning rate as the proxy:

Proxy → TargetMethodLR (Both)Accuracy (%) (Origin → Transfer)Δ\Delta Accuracy
ViT-B/32 → ViT-L/14Task Arithmetic1e-576.0 → 76.00.0
3e-568.9 → 68.90.0
AdaMerging1e-575.7 → 76.6+0.9
3e-568.0 → 66.5−1.5
ViT-B/16 → ViT-L/14Task Arithmetic1e-576.0 → 76.00.0
3e-568.9 → 68.90.0
AdaMerging1e-575.7 → 76.1+0.4
3e-568.0 → 65.0−3.0

In the matched setting, all model sizes consistently select $ \alpha=0.1 $ under lr 3e-5, and Task Arithmetic transfers without accuracy drop. This indicates that the optimal coefficients depend on the fine-tuning recipe but shift consistently across scales when the recipe is shared. The drop in the fixed-target setting therefore arises mainly from the recipe mismatch between proxy and target, rather than from the learning rate itself.

In our main experiments, the fine-tuning configuration of each checkpoint is selected independently (Appendix A), reflecting realistic scenarios where checkpoints are often fine-tuned spearately. While most selected checkpoints happen to be using the same lr 1e-5, some are using 3e-5, which may contribute to the losses in Table 3. In practice, transfer is more reliable when the proxy is fine-tuned with the same recipe as the target and is sufficiently trained. We will include these results in the final version.

AuthorsOpenReview ↗

Response (2/2)

Question 3

Our reported speedups assume that suitable proxy experts are already available, and only measure the cost of coefficient search. When proxy experts must be created, the additional cost is a one-time fine-tuning of a much smaller model (e.g., CLIP ViT-B/32 with 88M parameters for ViT-L/14 with 304M). As shown in our response to Question 2, the proxy can reuse the target’s fine-tuning recipe, avoiding a separate hyperparameter search.

Beyond compute, searching on smaller proxies also substantially reduces memory requirements. Optimization-based methods such as AdaMerging require large VRAM to hold all experts simultaneously, which can be prohibitive for large models. α\alphaTransfer makes these methods feasible on hardware that cannot accommodate the full set of target experts.

Reviewer_S5WhOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Optimization of merging coefficients in model merging is expensive, and especially so for large models and high number of tasks. The paper hypothesizes that the parameters can be learnt on smaller models of a model family and used directly on larger models. The consider three variations of model merging and the way they use coefficients and propose how to transfer coefficients in each case. For global and task-wise coefficients the transfer is direct and same coefficients are used, while for layer-wise coefficient the authors build on related literature and normalize coefficients of each layer based on the depth of the layer within networks such that depending on the relative depth the same coefficients are used across both smaller and bigger models. Further, interpolating coefficients is considered in the last case. The experiments across two vision model families, and one language family show a significant speedup and much lower memory usage compared to traditional evaluated-based and gradient-based methods. Moreover, adjusted for compute the proposed method even outperforms coefficient tuning directly on the target model.

Critical strengths
  • The writing is exceptional, the idea is simple and clearly presented, the problem is both well motivated through putting shortcomings of existing approaches into perspective, and the underlying intuition is justified through the experiments. Extensions of the base idea are simple, natural, and well supported by the literature. The empirical suite covers all the important bases and does not go into redundant experimentation. Overall, I feel the paper is simple enough to understand, while being complete and clear. The best illustration of this is as I was reading it I was jotting questions and by the end all my questions were answered. I feel the paper could be easily understood by an advanced high school student how well it is written, and please accept this as my commendation.
  • The performance benefits are impressive, to the point that makes me question their validity such as the top row of Figure 3. So I hope the authors will release the code if the paper is accepted so they can be properly verified by the independent community.
  • I very much like how layer-wise transfer is incorporated, as well as that the authors considered looking into the hybrid search.
Critical weaknesses

My main concern is how much use-cases this approach sacrifices compared to regular coefficient search. Namely, this work requires the existence of fine-tuned checkpoints on same datasets across both small and big models of the family. While this is not a limiting factor when we have access to task data and resources to do the fine-tuning, in reality many applications of model merging in practice may be on arbitrary checkpoints found in large model repositories published on the internet where access to the data is not possible and where fine-tuned checkpoints exist only for specific models within the family.

Minor, but some well-performing recent approaches such as WUDI and TSV-M are missing. What's the justification for omitting them? Even if they are omitted I believe there is value from the reader's point into knowing which approaches are compatible with the proposed solution and which aren't. On that note I would like to see Table 1 extended for more approaches and mark their compatibility with α\alpha-transfer.

Questions for authors
  • Are authors committed to release the code of their implementation if the paper is accepted?
  • I like Table 2 however, Spearman's ρ\rho is not particularly informative in isolation, do the authors believe these values could be put into perspective by including the same correlation metric for models of different family but roughly the same size as a form of "control"?
  • Please justify the omission of WUDI and TSV-M (or just general recent well performing works) from the paper (refer to weaknesses section).

Clarifying questions:

  • Table 4 shows significantly lower time for Qwen3 compared to vision models that are 10-100x smaller in Table 3. Is this a result of using PEFT on Qwen3 that was not mentioned or the number of tasks affect the runtime more significantly compared to the size of the model? On this note, has α\alpha-transfer been tested on any PEFT approaches, do the results hold?

Minor notes:

  • In Section 4, "Experimental Setup" listing all models and their respective size in the number of parameters in a table would marginally improve the following of the paper.
  • Table 1 overflows.
3 comments

We sincerely thank the reviewer for the positive assessment and for carefully reading our paper. Our response is as follows:

Weakness 1

We acknowledge that our method additionally requires fine-tuned checkpoints of a smaller model from the same family. In practice, however, this requirement is often satisfied, since developers commonly apply the same fine-tuning recipe across multiple model sizes, either to run pilot studies on smaller models before scaling up or to serve users with different compute budgets. For example, the WizardLM and Qwen2.5 families both release task-specific fine-tuned models (e.g., instruction following, math, and code) at several sizes. Settings where only a large fine-tuned checkpoint is available remain outside our current scope, and we leave them to future work.

Weakness 2 & Question 3

We thank the reviewer for this suggestion.

WUDI-Merging is designed to be coefficient-free. It obtains the merged weights by optimizing the merged task vector of each linear layer via gradient descent and adding it back to the pre-trained model without multiplying a scaling coefficient. Since no coefficient search is involved, α\alphaTransfer is not directly applicable.

In contrast, TSV-M applies a global scaling coefficient α\alpha to the merged task vector, and is therefore compatible with α\alphaTransfer. We have updated Table 1 in the revision to include both methods and mark their compatibility with α\alphaTransfer.

The table below shows the results of applying Transfer to TSV-M. α\alphaTransfer achieves a 2.6$\times$ to 5.5$\times$ speedup and a 59% to 69% reduction in memory, with an accuracy drop of at most 1.35 points. We will include the full results of TSV-M in the final version.

Proxy → TargetAccuracy (Origin → Transfer)Time (h) (Origin → Transfer)SpeedupMemory (GB) (Origin → Transfer)Reduction
ViT-B/32 → ViT-L/1486.29 → 85.746.96 → 1.285.46×2.12 → 0.6668.7%
ViT-B/16 → ViT-L/1486.29 → 84.946.96 → 1.853.77×2.12 → 0.8758.9%
SigLIP-Base → SigLIP-Large86.69 → 86.696.14 → 2.332.63×2.84 → 1.0762.2%

Question 1

Yes, we will release the code.

Question 2

As suggested, we added size-matched cross-family pairs as a control:

GroupModel PairSpearman ρ\rho
Same familyViT-B/16 – ViT-L/140.84
Same familyViT-B/32 – ViT-L/140.58
Same familySigLIP-Base – SigLIP-Large0.79
Cross familyViT-B/16 – SigLIP-Base0.83
Cross familyViT-B/32 – SigLIP-Base0.56
Cross familyViT-L/14 – SigLIP-Large0.74

The results show that cross-family correlations fall in the same range as same-family ones, indicating that the similarity of coefficient rankings is not determined by model family alone. We hypothesize that this is because all model pairs are evaluated on the same set of coefficients, and some task-level properties affect all models in a similar way. For example, configurations with a large sum of coefficients or a single dominant task coefficient tend to perform poorly regardless of the model. Moreover, both families use ViT backbones with contrastive image-text pre-training, which may further contribute to the similarity.

These results are consistent with our main claim, as coefficient rankings remain well correlated within each family. Interestingly, they also suggest that α\alphaTransfer may extend beyond a single family. We focus on same-family transfer as the most practical setting, and leave a systematic study of cross-family transfer to future work. We thank the reviewer for this suggestion, which provides a useful perspective on our results.

Clarifying question

We did not use PEFT in our experiments. The runtime difference mainly comes from the evaluation set size, which dominates the search cost. The vision benchmark has 83,829 evaluation samples across 8 tasks, while the language benchmark has only 3,555 samples across 4 tasks. We have not yet evaluated α\alphaTransfer on PEFT approaches, which is an interesting direction for future work.

Minor Notes

We thank the reviewer for the helpful notes. We have fixed the overflow of Table 1 in the revision, and will add a table listing all models and their parameter counts in Section 4.

Thank you very much for a detailed response. All my points have been addressed, and I am happy to keep my positive assessment. Congratulations on an excellent work, I look forward to seeing its impact on the community!

Thank you for the encouraging feedback! Your suggestions on recent merging methods and the cross-family control were very helpful. We have extended Table 1 accordingly, and will add the TSV-M results and the discussion on proxy availability to the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear Reviewers and AC,

We sincerely thank the Area Chair and all reviewers for their careful reading and valuable, constructive feedback. The discussion helped us substantially strengthen the paper. In particular, we:

  • Characterized when transfer is reliable through a controlled analysis that fixes the target experts and varies the proxy’s fine-tuning recipe. Transfer is robust to the proxy’s training duration once it is reasonably trained, while a learning rate mismatch between proxy and target can reduce performance. When the recipe is shared, the optimal coefficient shifts consistently across scales and transfer is preserved. This yields a practical guideline for selecting proxy experts (Reviewer cd6N).

  • Extended α\alphaTransfer to recent subspace-based methods, including TSV-M and Subspace Boosting, achieving 2.6–5.6$\times$ speedups and 59–69% memory reductions while largely preserving accuracy (Reviewers S5Wh and qrPx).

  • Added cross-family correlation experiments as a control, revealing that coefficient rankings can exhibit comparable correlations across different model families. This suggests that the coefficient landscape is largely shaped by the tasks, which is consistent with our within-family findings and indicates potential broader applicability (Reviewers S5Wh and qrPx).

Beyond the additional experiments, we also clarified the real world use-case and limitations of α\alphaTransfer, particularly regarding the availability and training costs of proxy checkpoints (Reviewers S5Wh and cd6N). We also expanded the discussion of compatibility with recent merging methods, now marked in the extended Table 1 (Reviewers S5Wh and qrPx). We also corrected a reporting error in Table 10, which does not affect any conclusions (Reviewer cd6N), and improved the presentation based on the reviewers’ suggestions (Reviewer S5Wh and qrPx).

We are grateful for the constructive discussion and will incorporate all of these experiments, analyses, and clarifications into the revised manuscript.

Best, Authors of Submission 38430

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.