acceptodds
Under review as a conference paper at ICLR 2027

A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic

Abstract

Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Task Arithmetic, Multi-task Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m5sSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper re examines the implicit subspace constraints introduced by task arithmetic in model merging. The authors points out that traditional methods limit candidate models to the subspace formed by task vectors by optimizing the combination coefficients of task vectors, which may hinder the discovery of better multi task weights. To verify this viewpoint, the paper compared coefficient optimization within the subspace with direct weight optimization, and further analyzed the effectiveness of searching outside the subspace through gradient update decomposition and S-RandOpt. The experiment covers various visual and language models and representative merging algorithms, and the results show that using auxiliary data for weight optimization can usually achieve better multitasking performance.

Critical strengths
  1. The paper re examines the traditional model merging process from the perspective of implicit subspace constraints, and the research problem has certain novelty and universal significance, providing valuable ideas for further exploring optimization methods for tasks beyond quantum space.

  2. The experiment covers multiple model architectures, tasks, and merging methods, and further analyzes subspace gradient updates, auxiliary data scale, and computational costs. The evidence is sufficient to support the main observation that weight space optimization has significant advantages under the studied setting.

Critical weaknesses
  1. The main performance improvement of the paper comes from using labeled auxiliary data to perform additional weight fine-tuning on the merged model. However, this process differs significantly from traditional coefficient search in terms of trainable parameter size and optimization capability. Therefore, existing experiments have not fully distinguished whether performance gains come from lifting task constraints into quantum space or from additional supervised multi task training.

  2. The explanation of the advantages of optimizing outside the subspace in the paper is still insufficient. Due to the lower dimensionality of the task towards the quantum space compared to the complete parameter space, gradient updates are mainly distributed outside the subspace and may not necessarily have sufficient explanatory power. In addition, some conclusions are overly generalized, such as the more flexible hierarchical coefficient optimization in the appendix being superior to weight fine-tuning under certain settings. The author also refers to the best performance of the test set in the limited search method as Subspace Best, and needs to more accurately explain the meaning of this empirical benchmark.

Questions for authors
  1. Can the authors further compare different degrees of subspace constraints and weight optimization under the same auxiliary data and computational budget to distinguish the performance benefits brought by search space expansion and additional supervised training?

  2. Can the authors supplement the experiment on controlling the gradient inside and outside the subspace to update the norm, and explain why the direction outside the subspace can produce better generalization performance? In addition, could you clarify the selection method of Subspace Best and discuss more comprehensively the situation where optimizing hierarchical coefficients is superior to fine-tuning weights?

3 comments

Thank you for your review. Please refer to our global response before reading our responses to your specific concerns and questions. We address your concerns below.

Weakness 1.1: Different from traditional coefficient search

This process (weight fine-tuning) differs significantly from traditional coefficient search in terms of trainable parameter size and optimization capability.

As discussed in the paper, we challenge the implicit regularization induced by coefficient search. We point out that under the existing evaluation scenario, the traditional coefficient search methods are actually underparameterized.

Weakness 1.2: Distinguish the performance gain

Therefore, existing experiments have not fully distinguished whether performance gains come from lifting task constraints into quantum space or from additional supervised multi task training.

Please see the global response for why the question is unclear. However, if the question is “whether gains come from removing the regularization or additional training”, the answer is the former, for two reasons:

  • As shown in Section 5.1, Subspace GD is a supervised multi-task training that spends more time and compute, but has worse performance than Weight GD due to the implicit regularization.
  • Outperforming the best performance within the subspace (Subspace-best) does not rely solely on GD. S-RandOpt in Section 4.2 demonstrates that sampling can also achieve better performance.

Weakness 2.1: Explanation of gradient descent performance gain

The explanation of the advantages of optimizing outside the subspace in the paper is still insufficient. … gradient updates are mainly distributed outside the subspace and may not necessarily have sufficient explanatory power.

To clarify, Section 4.1 does not try to explain which direction is better. It aims to visualize that gradient descent indeed leaves the task-vector subspace. Our explanation for gradient descent's superior performance in standard evaluation settings is the removal of implicit regularization.

Weakness 2.2 & Question 2.3: When is coefficient optimization better than weight fine-tuning?

In addition, some conclusions are overly generalized, such as the more flexible hierarchical coefficient optimization in the appendix being superior to weight fine-tuning under certain settings.

… discuss more comprehensively the situation where optimizing hierarchical coefficients is superior to fine-tuning weights?

  • We are not sure which conclusions the reviewer considers overly generalized. However, the paper's main finding is that the appropriate regularization strength depends on the auxiliary data, rather than that weight GD is always the best approach.
  • Coefficient optimization induces strong regularization, which can be beneficial in certain scenarios, as discussed in the paper. The bias-variance tradeoff is key to understanding when coefficient optimization is superior to fine-tuning weights. Please see the global response for further discussion.

Weakness 2.3 & Question 2.2: Subspace Best

The author also refers to the best performance of the test set in the limited search method as Subspace Best, and needs to more accurately explain the meaning of this empirical benchmark.

We summarize Appendix C.1 here for reference: “Subspace Best reports the highest test performance among the evaluated coefficient optimization methods (coefficient search, Subspace GD and Bayesian optimization), assessing the performance achievable within the task-vector subspace.”

Question 1: More comparisons

Can the authors further compare different degrees of subspace constraints and weight optimization under the same auxiliary data and computational budget to distinguish the performance benefits brought by search space expansion and additional supervised training?

We summarize the analyses from the paper here for reference:

  • Subspace GD vs Weight GD in standard evaluation (Section 3 & Section 5.1): using the same target, auxiliary data, and learning algorithm, subspace GD optimizes the model weights with implicit regularization. Even with more computational budget, subspace GD still performs worse.
  • Comparison in 2-task setting (Section 5.3): same as the standard evaluation except the data are scarce; optimizing with implicit regularization can have better test set performance.

The new experiment in Section 5.4 also addresses your question; please see the global response for more details.

Question 2.1: Geometric analysis

Can the authors supplement the experiment on controlling the gradient inside and outside the subspace to update the norm, and explain why the direction outside the subspace can produce better generalization performance?

As clarified earlier, we do not investigate which directions outside the task-vector subspace are better or why. Moreover, Section 4 does not claim that gradient descent achieves better generalization performance, nor is this claim true.

Thank you to the authors for their response. I have read and considered the authors’ response and have decided to maintain my original score.

We thank the reviewer for the quick response. We are confident that our responses have addressed the concerns and questions raised by the reviewer, including:

  • Clarify the main contributions of the paper
  • Clarify the performance gain of gradient descent
  • Clarify the findings of coefficient optimization versus weight fine-tuning
  • Clarify the Subspace Best estimation
  • Clarify the message of Section 4

We would also appreciate further clarification on a few points raised in the original review, as we want to ensure that we have accurately understood and addressed these concerns.

Weakness 1

Therefore, existing experiments have not fully distinguished whether performance gains come from lifting task constraints into quantum space …

Weakness 2

Due to the lower dimensionality of the task towards the quantum space compared to the complete parameter space, ...

In addition, some conclusions are overly generalized ...

We would be happy to provide further explanations or clarifications if needed. If we have successfully addressed your questions and concerns as we understand them, and the reviewer has no remaining concerns, we respectfully ask the reviewer to reconsider their assessment accordingly. Thank you again for your service.

Reviewer_sj2hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks how auxiliary data should be used in model merging: to select task-vector coefficients, or to adapt the merged weights directly. It formulates coefficient optimization as a restricted subspace search, compares full-weight adaptation for vision models and LoRA adaptation for language models, and studies the restriction through update decomposition and S-RandOpt. Results generally favor adaptation after merging. Further analyses examine initialization, data availability, and test-time adaptation.

Critical strengths
  1. When auxiliary data are already available, comparing coefficient selection with weight adaptation is appropriate. Distinguishing a merged model as a final result from its use as an initialization also helps clarify the role of merging in practice.
  2. The main observation is examined across merging methods and vision and language models, with analyses of initialization and data availability. The low-data, two-task results in §5.3 also show that stronger constraints can remain useful, helping establish the limits of the finding.
Critical weaknesses
  1. The main TIES and TSV-M parameterizations in Appendices B.1/C.1 are one-dimensional, whereas weight adaptation has substantially more freedom. This shows that the coefficient search is restrictive, but does not distinguish too few global coefficients from a limitation of the task-vector directions. Without layer-wise coefficients under the same objective, I find it difficult to identify the insight beyond familiar limitations of low-dimensional parameterization.
  2. Section 4.1 shows that optimized updates use components outside the task-vector span, but does not explain which directions are inadequate or why. S-RandOpt supports expansion along gradient-informed directions, but does not separate the value of that information from additional search freedom. LoRA often outperforming full-weight adaptation further suggests that greater freedom alone is not an adequate explanation.
  3. Test-time adaptation methods use different objectives, and layer-wise DivMerge outperforms GD on ViT-L/14 in Table 20, limiting the scope of the adaptation advantage. Appendix C.2 states that Figure 2 uses 1/T initialization for TA/DARE, but the Qwen3-1.7B values 81.48/81.03 in Table 17 match the 1.0 initialization in Table 16. This inconsistency makes the relevant results difficult to reproduce precisely.
Questions for authors
  1. Is the main limitation too few global coefficients or the task-vector directions themselves? Which results distinguish these explanations and account for LoRA outperforming full-weight adaptation?

  2. How do layer-wise coefficients compare with LoRA or full-weight adaptation under the same auxiliary data, supervised objective, and initialization? This comparison would directly affect my assessment of the central insight.

  3. Which TA/DARE initializations were used in Figure 2 and Table 17, and how do they correspond to Appendix C.2 and Table 16? How much of the test-time adaptation conclusion depends on differences in objectives?

3 comments

Thank you for your review. Please refer to our global response before reading our responses to your specific concerns and questions. We address your concerns below.

Weakness 1.1 & Question 1: Limitation of task-vector directions

(The main TIES and TSV-M parameterizations.)... This shows that the coefficient search is restrictive, but does not distinguish too few global coefficients from a limitation of the task-vector directions.

Is the main limitation too few global coefficients or the task-vector directions themselves? Which results distinguish these explanations and account for LoRA outperforming full-weight adaptation?

  • First, we believe that it is impossible to answer the request: “distinguish too few global coefficients from a limitation of the task-vector directions”. Adding more coefficients will add more directions, which makes it impossible to distinguish them.
  • Second, this request is not central to the paper. Our paper is asking: “To better use the auxiliary data, is the implicit regularization induced by coefficient optimization necessary?” Our goal is not to investigate the usefulness of task-vector directions, but rather to study the final step of model merging, which performs a linear combination of task vectors.
  • Third, the fact that LoRA outperforms full-weight adaptation is a direct result of the bias-variance tradeoff.

Weakness 1.2 & Question 2: Comparison with layer-wise merging

Without layer-wise coefficients under the same objective, I find it difficult to identify the insight beyond familiar limitations of low-dimensional parameterization.

How do layer-wise coefficients compare with LoRA or full-weight adaptation under the same auxiliary data, supervised objective, and initialization? This comparison would directly affect my assessment of the central insight.

New results in Section 5.4 address your questions.

  • Yes, low-dimensional parameterization is known to underfit. However, this paper is the first to identify this form of implicit regularization induced by coefficient optimization and systematically analyze it.
  • The central insight is that regularization strength affects generalization. In existing benchmarks with abundant auxiliary data, implicit regularization induced by coefficient optimization limits test-set performance; in scenarios with scarce auxiliary data, appropriate regularization leads to better test-set performance.

Weakness 2.1: Section 4 is not enough

Section 4.1 shows that optimized updates use components outside the task-vector span, but does not explain which directions are inadequate or why. S-RandOpt supports expansion along gradient-informed directions, but does not separate the value of that information from additional search freedom.

As we mentioned in the earlier response, whether directions are inadequate is not the question that we aim to answer. Also, S-RandOpt aims to show that finding better multi-task weights beyond the task-vector subspace does not rely on gradient descent. To explore beyond a subspace, one has to add additional degrees of freedom.

Weakness 2.2 & Weakness 3.1: Full-weight GD is not always the best

LoRA often outperforming full-weight adaptation further suggests that greater freedom alone is not an adequate explanation.

…, and layer-wise DivMerge outperforms GD on ViT-L/14 in Table 20, limiting the scope of the adaptation advantage.

Your observations are correct, but these experimental results support the paper’s main finding rather than reveal a weakness. As stated in Appendix D.4, the bias-variance tradeoff is the key consideration when choosing the regularization strength. We did not say that full-weight adaptation is always the best.

Weakness 3.2 and Question 3.1: Initialization points

Appendix C.2 states that Figure 2 uses 1/T initialization for TA/DARE, but the Qwen3-1.7B values 81.48/81.03 in Table 17 match the 1.0 initialization in Table 16.

Which TA/DARE initializations were used in Figure 2 and Table 17?

As we stated in Section 3.1.1, the Naive are the best between 1/T and 1.0, and we initialize weight-space GD from this naive baseline. For TA/DARE of LLMs, all are initialized from 1/T except Qwen3-1.7B, where 1.0 is the better coefficient. We have updated Appendix C.2 in the revision to clarify this.

Question 3.2: Test-time adaptation’s objective

How much of the test-time adaptation conclusion depends on differences in objectives?

Given the two-task setting in Section 5.3, the comparison between LoRA and full-weight fine-tuning in Appendix C.2, and the new results in Section 5.4, we believe there is sufficient evidence to support the paper’s main finding. A comparison with existing test-time adaptation methods for model merging serves as an additional demonstration, and changing their objectives would make them different methods. If the reviewer believes there are reasons to compare these objectives, we are happy to conduct experiments.

Thank you for the author's response; the additional experiments have addressed some of my concerns. My primary reservation regarding this work is that—beyond the established understanding that "stronger constraints can be beneficial when data is scarce"—the paper offers limited practical guidance: specifically, when should auxiliary data be used to determine the model-merging strategy, and when should it be used to directly fine-tune the merged model? Consequently, I will maintain my current rating.

Dear Reviewer s2jh,

Thank you for the response. We now address your remaining concerns.


beyond the established understanding that "stronger constraints can be beneficial when data is scarce"—the paper offers limited practical guidance

  • The model merging literature treats coefficient optimization as the method itself, not as a choice of regularization strength. Therefore, our key contribution is to recognize coefficient optimization as a form of regularization and place it within a broader framework of optimization strategies with varying degrees of flexibility. We further connect this framework to the classic bias-variance tradeoff in machine learning, suggesting that “stronger constraints can be beneficial when data are scarce”. The bias-variance tradeoff indicates that no single solution is optimal across all scenarios.
  • Without recognizing the existing regularization induced by task arithmetic, practitioners may continue to rely on coefficient optimization even when sufficient auxiliary data are available to benefit from more flexible fine-tuning, which is the case in standard model merging benchmarks.

specifically, when should auxiliary data be used to determine the model-merging strategy, and when should it be used to directly fine-tune the merged model?

We would like to clarify the scope of our research question. We compare two ways of leveraging the same auxiliary data: (1) coefficient optimization and (2) directly fine-tuning the merged model. The results show that:

  • (Section 5.4) The best method is data-dependent.
  • (Section 3 and Section 5.5) When the data is abundant, as in standard model merging benchmarks, directly fine-tuning the merged model performs generally better than coefficient optimization.

We would be happy to provide further explanations or clarifications if needed. Thank you again for your service.

Reviewer_HikhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper suggests that coefficient optimization in task arithmetic restricts the model within a constrained task-vector subspace. It demonstrates across various vision and language benchmarks that directly optimizing the weights of the merged model using auxiliary data yields better performance. Furthermore, it analyzes gradient updates outside the subspace and the associated bias-variance trade-off.

Critical strengths
  1. By analyzing the in-subspace and out-of-subspace components of gradient updates, the paper shows that updates in out-of-subspace directions are crucial for performance improvement.

  2. The results are validated across multiple model architectures and merging methods, alongside an analysis of the generalization trade-off relative to the scale of auxiliary data.

Critical weaknesses
  1. Surgery [1] and ProbSurgery [2] train a separate representation correction module, while SyMerge [3] jointly optimizes merging coefficients and task-specific layers. These methods also leverage richer expressiveness than coefficient-only optimization. Although the authors acknowledge that backbone weight optimization is structurally distinct, comparisons demonstrating practical advantages and novelty over these prior approaches are lacking.

  2. Weight GD utilizes labeled auxiliary data and a significantly larger number of trainable parameters. Consequently, it remains unclear whether the performance gains stem from mitigating subspace constraints or simply from the extra degrees of freedom and supervision during training. Notably, in Table 20, layer-wise DivMerge achieves higher performance than Weight GD on ViT-L/14.

[1] Representation Surgery for Multi-Task Model Merging (ICML2024)

[2] Representation Surgery in Model Merging with Probabilistic Modeling (ICML2025)

[3] SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation (ICML2026)

Questions for authors
  1. Can you provide a comparison against Surgery, ProbSurgery, and SyMerge under the exact same auxiliary-data budget? What is the unique advantage of backbone weight optimization?

  2. Do the performance improvements hold under identical supervision as well as similar trainable-parameter and compute budgets?

  3. How were classes with insufficient validation samples handled during the data scaling experiments?

3 comments

Thank you for your review. Please refer to our global response before reading our responses to your specific concerns and questions. We address your concerns below.

Weakness 1 & Question 1: Comparison with related works

Surgery and ProbSurgery train a separate representation correction module, while SyMerge jointly optimizes merging coefficients and task-specific layers.

We thank the reviewer for pointing out these relevant works. As clarified in the global response, our contribution is a broader view of how auxiliary data should be used in model merging, not a new method. Weight GD is just an example to show that simply increasing model complexity improves performance on standard benchmarks.

As requested, we compare Weight GD with these related works. The results are shown in Table R1. We will include these results in the revision.

Table R1: Comparison with related works for ViT-B-32 in the 9-task setting’s test-time adaptation. Under the standard benchmark, Weight GD has the best performance.

Merging InitOriginalSurgeryProbSurgeryWeight GD
Task Arithmetic62.4177.0381.6382.46
TIES-Merging67.3578.5181.4483.64
AdaMerging layerwise75.6682.0283.2284.88
SyMerge83.83———

Weakness 2.1 & Question 2: Source of weight GD’s gain

Weight GD utilizes labeled auxiliary data and a significantly larger number of trainable parameters. Consequently, it remains unclear whether the performance gains stem from mitigating subspace constraints or simply from the extra degrees of freedom and supervision during training.

We respectfully note that mitigating the subspace constraint and gaining extra degrees of freedom are the same thing. Coefficient-based merging restricts the merged weights to the span of the task vectors, so leaving this subspace necessarily requires more trainable parameters.

Do the performance improvements hold under identical supervision as well as similar trainable-parameter and compute budgets?

For Question 2, the degrees of freedom (number of trainable parameters) are exactly the variable our experiments control and study, ranging from global-wise merging to full-weight fine-tuning (see global response and section 5.4 in the revised version). Constraining Weight GD to a parameter budget similar to that of coefficient-based merging would reintroduce another strong regularization we show to be unnecessary. The point is that an appropriate parameter budget should be determined by the amount of auxiliary data.

Weakness 2.2: Weight GD is not always the best

Notably, in Table 20, layer-wise DivMerge achieves higher performance than Weight GD on ViT-L/14.

The reviewer’s observation is correct, but this result supports our main finding rather than revealing a weakness. Gradient descent on larger models like ViT-L/14 has more trainable parameters, making it more prone to overfitting. In contrast, layer-wise DivMerge provides implicit regularization that reduces the risk of overfitting.

Question 3: Details of data scaling exp

How were classes with insufficient validation samples handled during the data scaling experiments?

For classes containing insufficient validation samples, we use all available samples without replacement. We have clarified this in Appendix C.3 in the revision.

Thank you for the detailed response. I appreciate the authors' clarifications. However, I remain unconvinced that the proposed perspective offers sufficiently distinct insights beyond the known effects of model capacity and regularization. Furthermore the practical advantages over existing flexible merging methods remain unclear. Therefore I will maintain my original score.

Dear Reviewer Hikh,

Thank you for the quick response. We now address your remaining concerns.


However, I remain unconvinced that the proposed perspective offers sufficiently distinct insights beyond the known effects of model capacity and regularization.

We would like to clarify that while the bias-variance tradeoff itself is well known, this work connects it to model merging. The model merging literature treats coefficient optimization as the method itself, not as a choice of regularization strength.

Coefficient optimization is incorporated into our unified framework as an option, and this unification provides practical implications, as discussed in the following response.

Furthermore the practical advantages over existing flexible merging methods remain unclear.

By connecting the bias-variance tradeoff to model merging, the practitioner can choose the regularization strength based on auxiliary data:

  • (Section 3 and Section 5.5) When data are abundant, as in standard model merging benchmarks, directly fine-tuning the merged model performs better than coefficient optimization.
  • (Section 5.3 and Section 5.4) When data are scarce, coefficient optimization performs better due to stronger regularization.

Please note that our work is the first to provide this practical guideline for model merging.


We would be happy to provide further explanations or clarifications if needed. Thank you again for your service.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear AC and reviewers,

Thank you for handling our paper. In this response, we would like to address several misconceptions that appear across multiple reviews.

Paper’s main finding

The paper’s main contribution is a unified and broader look at model merging, not proposing a method. We identify that coefficient optimization under task arithmetic has an implicit regularization effect, and systematically analyze the tradeoff between regularization strength and optimization flexibility across multiple scenarios.

  • In existing benchmarks, where auxiliary data are abundant, we find that the coefficient optimization generally limits the model’s performance. Full-weight gradient descent is just a way to test this regularization. While gradient descent achieves superior performance in Section 3, our main contribution is not to propose gradient descent as a model merging method.
  • In general, the bias-variance tradeoff is the key principle for deciding how to use auxiliary data. The two-task setting in Section 5.3, the comparison between LoRA and full-weight fine-tuning in Appendix C.2, and the comparison between layer-wise merging and weight GD (full-weight fine-tuning) in Appendix D.4 support this conclusion. In these scenarios, full-weight fine-tuning may not achieve the best test-set performance as it is more prone to overfitting.
  • As we stated in the Conclusion, we did not identify a one-size-fits-all method for using auxiliary data. We did not say that full-weight fine-tuning is always the best.

New Experiment

To help the reader understand the main contribution of the paper, we conducted a new experiment in Section 5.4 to further demonstrate why the bias-variance tradeoff is the key principle.

Across several auxiliary data sizes, we use the number of trainable parameters as a proxy for model complexity to empirically measure the bias-variance tradeoff. Key findings:

  • With a small training dataset: more flexible methods (LoRA and full-weight fine-tuning) are more prone to overfitting (high variance), leading to higher test error. In contrast, less flexible methods (global, task, and layer-wise merging) have higher bias but achieve a better bias-variance tradeoff.
  • With a large training dataset: more flexible models achieve lower test error when more training data are available, as their variance decreases.

By identifying existing uses of auxiliary data, including global-, task-, and layer-wise merging, as forms of implicit regularization, our findings enable the community to rethink model merging from a bigger picture.

Responses to reviewers' comments

Reviewer Hikh - Weakness 2

Notably, in Table 20, layer-wise DivMerge achieves higher performance than Weight GD on ViT-L/14.

Table 20 supports our finding rather than revealing a weakness. As stated in Appendix D.4, the bias-variance tradeoff is the key consideration when choosing the regularization strength. Gradient descent on larger models like ViT-L/14 has more trainable parameters, making it more prone to overfitting. In contrast, layer-wise DivMerge provides implicit regularization that reduces the risk of overfitting.

Reviewer sj2h - Strength 2

The low-data, two-task results in §5.3 also show that stronger constraints can remain useful, helping establish the limits of the finding.

We respectfully clarify that these results are not a limit of the finding; they ARE the finding. Our finding is that generalization is the key factor in determining regularization strength, rather than “full-weight fine-tuning is always the best”.

Reviewer m5sS - Weakness 1&2

Therefore, existing experiments have not fully distinguished whether performance gains come from lifting task constraints into quantum space or from additional supervised multi task training.

Due to the lower dimensionality of the task towards the quantum space compared to the complete parameter space, gradient updates are mainly distributed outside the subspace…

Several statements in this review are difficult to interpret. The key term “quantum space” does not appear in our paper or the model merging literature, and phrases such as “lifting task constraints into quantum space” and “dimensionality of the task towards the quantum space” do not correspond to any concept we are aware of. We have responded under our best interpretation in the individual response, and we would appreciate it if the reviewer could clarify the intended meaning of these two points, so that we can address them more precisely.

Revision

To help readers more easily grasp the paper's key contributions, we made minor revisions to the Abstract, Introduction, Section 4.1, and Conclusion. The revisions are highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.