- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the problem of finding good merging coefficients, i.e., how to weigh different task vectors, in model merging. Specifically, the paper proposes to search for optimal merging coefficients using a smaller model variant from the same model family and then transferring the coefficients to a larger model. In this way, expensive search on the larger model can be avoided, which allows either less compute requirements overall, or more evaluations, i.e., potentially better coefficients, with the same compute budget. To account for different model depths, the paper proposes to match coefficients at the same relative depth and interpolate them. Experiments on image-test encoders and large language models confirm the method works, i.e., the coefficients transfer, and actual compute savings can be achieved.
Critical strengths
Generally, I find the paper very clear and to-the-point:
(S1) The problem and the solution are intuitive and well-motivated, but not trivial, the experiments are sufficient to support the claims, and the paper also considers potential obstacles, like different model depth. The supplementary material reports the hyperparameters and complete results in detail, which I found helpful. While reading, the ablation in Sec 4.5 (using the transferred coefficients as initialization for coefficient search on the larger model) came to my mind early on, so it was nice this was actually included.
(S2) The paper addresses a real and practically impactful problem by a natural solution that can be easily implemented. Here, I like that the solution does not introduce architectural complexity or complicated methods. Therefore, I expect this finding will be helpful in practice, so communicating it via ICLR is useful and beneficial to the community.
Critical weaknesses
While I find this paper clearly meets the bar for acceptance, my largest concern is about transfer to other merging methods beyond task vector averaging. For example, recent developments include subspace projections in weight [1, 2] and activation space [3], where different hyperparameters apply. Task Arithmetic and variants are common, and they are simple and practical, so it's reasonable to develop methods tailored to these, but it remain open how future-proof the paper is. This is relevant, because it may reduce the impact of the paper.
References:
[1] Gargiulo et al. "Task singular vectors: Reducing task interference in model merging." CVPR 2025
[2] Skorobogat et al. "Subspace-boosted model merging." arXiv 2025
[3] Wei et al. "SSR-Merge: Subspace Signal Routing for Training-Free LoRA Merging in Diffusion Models." ICML 2026
Questions for authors
(Q1) Do the authors think, or have evidence, that hyperparameter transfer also works for other merging parameters than task vector coefficients, e.g., for the subspace methods cited above? I would be very curious about a discussion of this aspect.
(Q2) While reading, I was also very curious about the limitations. In particular, the paper already mentions
may alter the optimization landscape, making direct coefficient mapping less effective
I would be curious what are the factors that lead to this. For example, we know that LLMs of the same family behave similarly, because they are trained on similar data with similar settings. However, how far can models be apart to still support coefficient transfer? The paper also mentions "cross architectures" (line 134), which probably refers to models with different depth, but I would be curious if this also works for models that have slightly different components, such as linear attention instead of standard attention. In summary, I'm interested in whether (especially task-wise) coefficients depend more on the (task) data or the model.
Perhaps the authors could compare task-wise coefficients across architectures?
Weakness 1
We thank the reviewer for the positive assessment and for pointing us to these recent subspace-based methods. To evaluate whether αTransfer extends beyond task-vector averaging, we applied it to TSV-M and to Subspace Boosting on top of Task Arithmetic and TIES-Merging:
Across all settings, αTransfer achieves a 2.6$\times$ to 5.6 × speedup and a 59% to 69% memory reduction. The accuracy is preserved within 1.35 points in all cases except Subspace Boosting (TIES) on ViT, where the drop is 2.57 points.
In contrast, activation-space methods such as SSR-Merge compute their merging components in closed form from activation statistics, requiring only a single pass over calibration data. As there is no costly search to accelerate, they fall outside the scope of αTransfer, though whether such statistics share structure across scales is an interesting direction for future work.
Question 1
We thank the reviewer for this interesting question. While we have not yet systematically studied this, we believe that transfer is likely to extend beyond merging coefficients. Some hyperparameters are defined in relative terms (e.g., the retained rank ratio in TSV-M) and are thus comparable across model sizes. Moreover, since models within the same family are fine-tuned on the same data, we hypothesize that the relative ranking of such hyperparameters is largely preserved across scales, as we observe for merging coefficients. We leave a systematic study of this direction to future work.
Question 2
We thank the reviewer for the thoughtful questions.
Clarification on “cross architectures.”
We use the phrase “cross architectures” in line 134 to express that coefficient transfer within a model family holds consistently across different architectures (ViT, Qwen3), rather than referring to models with different depth. We will revise the wording to avoid this confusion.
Dependence of coefficients on data or model
To investigate this question, we compared the Spearman correlation of coefficient rankings across model families:
The results show that cross-family correlations can also be as high as within-family correlations, suggesting that the coefficient landscape is largely shaped by the task data. This also indicates that αTransfer may be applicable even beyond a single model family, which we consider a promising direction for future work.
Regarding the differences in individual components such as linear attention, a controlled ablation would require pre-training models that differ only in that component under an identical setup. Since our experiments rely on publicly available pre-trained models, such an ablation is beyond our current scope.
Comparing task-wise coefficients
We note that directly comparing the optimal coefficients across models may not be informative, as the optimum found depends on the search algorithm. For example, on a non-convex landscape, optimization-based methods such as AdaMerging may converge to quite different points across models, even when the overall landscapes are similar. Indeed, we observe that the coefficients found on different models are often not close to each other, yet transfer remains effective. This suggests that what transfers is the overall shape of the landscape, under which coefficients found on the small model also perform well on the large model, rather than the exact location of the optimum.