acceptodds
Under review as a conference paper at ICLR 2027

SCouT: Softly Coupled Fine-Tuning for Preserving Specialization while Improving Mergeability

Abstract

The pre-training-fine-tuning paradigm is widely used to adapt large-scale models to downstream tasks, but maintaining separate task-specific models incurs substantial storage and deployment overhead. Model merging addresses this by combining multiple specialists into a single multitask model. However, independently optimized specialists can be poorly compatible, leading to a substantial drop in merged-model performance. We propose Softly Coupled Fine-Tuning (SCouT), which introduces merge-aware coupling during fine-tuning to improve mergeability while preserving task-specific performance. The coupling strength interpolates between independent fine-tuning and conventional multitask learning, recovering independent fine-tuning at zero coupling and approaching a shared multitask solution under strong coupling. Our analysis identifies a favorable intermediate regime in which weak coupling improves post-merge risk linearly in the coupling parameter while affecting specialist risk only quadratically, explaining why mergeability can improve much faster than specialization degrades. Experiments across model architectures and merging rules show that SCouT substantially improves post-merge performance while retaining strong specialists. These results suggest that weak cross-task coupling can steer fine-tuning toward more compatible solutions while still largely preserving their standalone specialization.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model merging, fine-tuning, multitask learning, transfer learning, deep learning, foundation models.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zRF8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to improve the mergeability of task specialists while preserving their individual performance. SCouT adds a parameter penalty between each specialist and a prescribed merged model during fine-tuning. A local analysis gives a conditional regime where merged risk improves to first order in the coupling strength while specialist risk changes to second order. Experiments with CLIP and RoBERTa show a useful tradeoff across coupling strengths and several post-hoc merging rules.

Critical strengths
  1. The empirical tradeoff is useful and well supported. Table 1 raises the ViT-B/32 merged accuracy from 61.66 to 81.27 while keeping specialist accuracy at 87.83, compared with 88.62 for independent fine-tuning. Table 2 shows gains across five merging rules, and Appendix D extends the result to GLUE. The paper also measures communication and memory costs in Appendix E. These results give practical evidence that mild coordination can retain useful specialists and improve their combined model.

  2. The paper connects its method to an interpretable local risk analysis. Section 4 states the positive alignment condition needed for the weak-coupling advantage, and Figures 3 and 4 examine both risk response and the broader performance tradeoff. This conditional explanation helps understand why mergeability can improve before specialist performance drops substantially.

Critical weaknesses
  1. I find the distinction from standard soft parameter sharing unclear in the implemented setting. Equation 16 uses the parameter mean as the merged model, so its coupling is exactly a pairwise squared-distance penalty up to a constant factor. Section 2.2 acknowledges this close connection. I do not see a new regularizer arising from the change in downstream goal alone, and I currently understand the main contribution as the merging perspective and its validation. I would find the originality easier to assess if the authors stated this equivalence explicitly.

  2. My concern with Theorem 1 and Appendix A.3 is how the stationary result relates to the actual optimizer. Equations 55 to 62 establish a zero projected gradient on the range of the independent specialists' Hessians. A nonzero component of the full gradient can still remain in the Hessian null space. Algorithm 1 does not impose the range constraint. I would like the authors to state the restricted stationary result explicitly and explain its connection to the unconstrained training trajectory. I regard this as a fixable issue in the scope of the claim, but the risk asymmetry is a central contribution and needs this connection.

Questions for authors
  1. I would like the authors to state the equivalence between the uniform isotropic setting in Equation 16 and pairwise soft parameter sharing, and explain which part of the contribution goes beyond that existing objective.

  2. I would like to know whether Theorem 1 concerns stationary points of the reduced objective on the Hessian range. I would also like the authors to identify the condition connecting this branch to Algorithm 1 when the Hessians have null spaces. A corrected statement would help resolve my concern.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zRF8

We thank the reviewer for the positive and constructive assessment, and for the precise technical comments, which helped us sharpen both the positioning and the theory of the paper. The first concern is shared with other reviewers and is addressed in detail in the General Response (GR); we summarize it below and answer the second concern, which is specific to this review, in full.

W1 / Q1: Equivalence with soft parameter sharing

The reviewer is correct in pointing out the equivalence with soft parameter sharing, and the revision now states this fact explicitly (Sec. 5.1, App. C.1). As detailed in GR, CC1, our contributions beyond this objective are (1) to show, to our knowledge for the first time, that the soft coupling/regularization makes specialists highly mergeable, and to characterize the resulting specialization-mergeability trade-off (Theorem 1); and (2) as the reviewer anticipated, SCouT can generalize to a family of merge-aware algorithms with general training-time merge operators and coupling matrices. These variants are evaluated in the revision (App. C.1-C.2).

W2 / Q2: Stationarity in Theorem 1 and connection to Algorithm 1

We thank the reviewer for this careful observation, which led us to revise Theorem 1 and to connect the theory more closely to the algorithm.

The revised Theorem 1 no longer needs the range restriction. Assuming positive definite Hessians Hk≻0H_k\succ0, Theorem 1 gives a unique stationary point of the unconstrained objective Fλ\mathcal F_\lambda,

ϑλ=ϑo−λH−1∇ϑC(ϑo)+O(λ2),\vartheta^\lambda = \vartheta^o - \lambda H^{-1} \nabla_\vartheta \mathcal C(\vartheta^o) + \mathcal O(\lambda^2), which is exactly the objective optimized by Algorithm 1. For singular Hessians Hk⪰0H_k\succeq0, a new remark (App. A.4) states the range-restricted result explicitly.

We acknowledge that neither result fully settles the case of singular Hessian HH: Theorem 1 requires a positive definite Hessian Hk≻0H_k\succ0, while the remark holds only on the range of HH. However, assuming an invertible (or damped) Hessian is standard in local perturbation analyses of deep networks, e.g., influence functions use H+γIH+\gamma I (Koh & Liang, ICML 2017; Grosse et al., 2023). More importantly, the contribution of Theorem 1 is not an exact formula for ϑλ\vartheta^\lambda, but the mechanism and scaling orders it reveals: weak coupling improves the merged risk at first order while costing specialist risk only at second order. These orders are what the paper relies on, and they are confirmed empirically (Fig 3, Fig 4 and Table 1).

Beyond the revised theorem, the revision strengthens the link between Theorem 1 and practical SCouT on two further fronts. First, theoretically, a new lemma (Lemma 1 in the main text) shows that, in the local quadratic model, the alignment coefficient χ\chi of Theorem 1, which governs the first-order post-merge improvement, is guaranteed to be positive under curvature-aware coupling. Second, empirically, Fig. 3 tests the predictions on checkpoints of Algorithm 1 (5 seeds): the post-merge risk decreases linearly, with a fitted slope giving χ≈9.9>0\chi\approx9.9>0, while the specialist risk grows only quadratically. The asymmetry predicted by Theorem 1 thus carries over to the actual training trajectory.


We hope these revisions address the reviewer's concerns. As the reviewer noted, soft coordination during fine-tuning can retain useful specialists while improving their merged model. We believe this opens a new direction: coordinated fine-tuning in settings where moderate communication is allowed, bridging model merging, multitask learning and distributed training. We hope it will be of broad interest to these communities, and we would be grateful if the reviewer would consider updating their score.

I appreciate the authors' detailed response and find the points well-reasoned. The clarifications have largely addressed my primary concerns, so I am pleased to maintain my score and lean towards recommending acceptance.

Reviewer_piYqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a new model merging method, motivated by the trade-off between preserving task-specific performance and improving model mergeability. Unlike conventional merge-aware fine-tuning methods that optimize individual specialists independently, SCouT introduces a coupling regularizer that encourages specialists to remain close to their training-time merged model. The coupling strength controls the trade-off between independent fine-tuning and hard multi-task learning. The paper further provides theoretical analyses showing that weak coupling can improve post-merge performance at first order while affecting specialist performance at second order. Experimental results on CLIP vision encoders and RoBERTa demonstrate substantial improvements in merging performance while largely preserving individual specialist accuracy.

Critical strengths
  • The proposed method is simple and provides a flexible way to control the specialization–mergeability trade-off.
  • The experimental results are promising, showing substantial improvements over independent fine-tuning and several existing merge-aware fine-tuning approaches. The paper also evaluates multiple merging operators and provides empirical analyses supporting the theoretical observations.
Critical weaknesses
  • Limited practical applicability

    • The proposed method, SCouT, assumes access to other task experts (or specialists) throughout the stages of fine-tuning of each specialist. To the reviewer, this seems to break the assumption/goal of model merging field and the practicality. Model merging is interested in finding a way to merge specialists that are independently fine-tuned from the same pretrained model. The proposed method seems more like a multi-task learning (MTL) instead of model merging work.
  • Concerns regarding novelty and originality

    • The main SCouT objective is closely related to conventional regularization-based soft parameter sharing. In particular, with isotropic coupling and uniform merging coefficients, the proposed regularizer is mathematically equivalent to a pairwise parameter-distance penalty among specialists. A direct comparison with conventional soft-MTL or consensus-regularization approaches is important for establishing the incremental contribution. Without these comparisons, it is difficult to determine whether the improvements arise from the proposed merge-aware formulation or simply from encouraging parameter similarity across task-specific models.
  • Insufficient analysis of the coupling mechanism

    • The main experiments use only isotropic coupling and uniform training-time merging. The effects of alternative coupling geometries, nonuniform task weights, and different coupling targets are not sufficiently investigated.
    • More importantly, comparisons with simpler alternatives, such as standard pairwise parameter regularization or regularization toward the pretrained initialization, would help determine whether the improvements specifically arise from the proposed merge-aware formulation or simply from keeping task-specific parameters closer together.
Questions for authors
  • The scope of model merging
    • The proposed method seems to be more suited for multi-task learning, rather than model merging. Could the authors clarify which practical model-merging scenarios SCouT is intended for and what advantages it offers over conventional multi-task learning in these settings?
  • Novelty relative to conventional soft parameter sharing.
    • Could the authors clarify what distinguishes the proposed optimization mechanism from existing regularization-based soft-MTL methods?
  • The gap between theoretical result and proposed method.
    • Corollary 1 assumes χ>0\chi>0, but this condition does not appear to be guaranteed by the proposed coupling objective. Can the authors clarify under what conditions this assumption holds, particularly for the isotropic coupling used in the experiments?
3 comments
AuthorsOpenReview ↗

Response to Reviewer piYq

We thank the reviewer for the careful review and for finding SCouT simple, flexible and empirically promising. Several concerns are shared with other reviewers and are addressed in the General Response (GR). Below, we answer the reviewer's concerns in turn.

W1 / Q1: Scope and practical applicability

The reviewer is correct in pointing out that when αk=1/K\alpha_k=1/K and Ak=IA_k=I, our formulation reduces to soft parameter sharing. The revision now states this explicitly immediately after the experimental setting (Sec. 5.1).

In GR, CC1, we clarify our contributions relative to the literature on soft parameter sharing and model merging. In summary, (1) we are, to our knowledge, the first to study the mergeability of such coupled models and to characterize the resulting specialization--mergeability trade-off, both empirically (Fig. 4) and theoretically (Theorem 1). (2) With general training-time merge operators and coupling matrices AkA_k, SCouT extends beyond standard soft parameter sharing to a family of merge-aware algorithms. The revision evaluates several of these variants (App. C.1-C.2): at some extra data or computation cost, they further improve performance, and we give a simple guide for choosing among them.

Moreover, as the reviewer notes, SCouT requires access to the other specialists during fine-tuning, which incurs additional communication to compute the merged model. As discussed in GR, CC2, this cost buys a large performance gain over independent fine-tuning (61.66% →\rightarrow 81.27% merged accuracy in Table 1). This communication cost can be reduced substantially with periodic synchronization of the merged model (App. C.3) that 20×20\times fewer exchanges cost only 1.1 points in merged accuracy (Table 6). These findings make SCouT well suited to distributed and federated settings with moderate communication budgets, where both strong specialists and a capable merged model are needed.

W2 / Q2: Novelty relative to soft parameter sharing

As discussed in W1/Q1, uniform, isotropic SCouT coincides with standard soft sharing. Our contribution lies in (1) establishing and characterizing this coupling as a tool for mergeability (Theorem 1), and (2) generalizing it to a family of merge-aware couplings with arbitrary training-time merge operators and coupling matrices (App. C).

Regarding a direct comparison with soft-MTL baselines: since uniform, isotropic SCouT is exactly pairwise ℓ2\ell_2-regularized soft MTL, this comparison is already given by our main SCouT rows (Table 1). To our knowledge, we are the first to evaluate this strategy both before and after merging, and to characterize the resulting trade-off between specialization and merge.

We agree that, in its simplified form, SCouT does directly encourage parameter similarity. However, similarity alone does not explain the gains. With various training-time merge operators (App. C.1) and curvature-aware coupling (App. C.2), post-merge performance improves further. What matters is therefore not how close the specialists are, but whether they are close where it affects the merged model, which is exactly what the merge-aware formulation targets.

W3: Insufficient analysis of the coupling mechanism

We thank the reviewer for this suggestion; this analysis was indeed missing from the original submission. The revision adds experiments on alternative training-time merge operators, including Fisher, RegMean and AdaMerging (App. C.1), and on curvature-aware coupling with a diagonal Fisher (App. C.2).

Regularization toward the pretrained model, as proposed by the reviewer, is also covered in App. C.3, which corresponds to periodic synchronization with the pretrained initialization as the merged model (ι=2000\iota=2000). It reaches only 56.66% post-merge accuracy. This shows that merely keeping the specialists close is not enough: the gains require exchanging information and coupling to the merged model.

Q3: Positivity of χ\chi

Indeed, χ>0\chi>0 is not automatic, since the coupling reduces the disagreement between the specialists and their merge, not the post-merge risk itself. GR, CC3 addresses this both theoretically and empirically. In short, a new lemma guarantees χ>0\chi>0 under curvature-aware coupling, and for the isotropic coupling used in our experiments, the fitted post-merge slope gives χ≈9.9>0\chi\approx9.9>0 (Fig. 3).


We hope these clarifications and new results address the reviewer's concerns. As the reviewer noted, SCouT offers a simple and flexible way to control the specialization-mergeability trade-off; we see its position between multitask learning and model merging as its key contribution. We would be grateful if the reviewer would reconsider the score in light of these revisions.

Thank you for the rebuttal. But, my concerns remain. The authors agree that the proposed method is similar to soft parameter sharing. And, this alone brings substantial performance improvement. Furthermore, independent specialists plus hard MTL already perform on par with the proposed method. Overall, I find the novelty and the effectiveness of the proposed method limited.

We thank the reviewer for the follow-up and would like to briefly clarify both points.

Novelty. Our point is precisely that soft sharing had not been recognized as a tool for mergeability. To our knowledge, no prior work has shown that coupling during fine-tuning substantially improves the merged model, or characterized the asymmetric response of specialization and mergeability (Theorem 1, Lemma 1). We believe this finding is valuable to the model merging, multitask learning and distributed training communities.

Independent specialists + hard MTL. SCouT combines the strengths of both: specialists close to independent FT and a merged model close to hard MTL. However, using the two baselines together does not offer the same, as it requires two separate training runs (about 2×2\times the cost of independent FT) and pooled data for hard MTL, whereas SCouT achieves both in a single run at without sharing data.

Reviewer_yrvqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCouT, a fine-tuning method that improves the compatibility of task-specific models before merging. It introduces a coupling regularizer that encourages specialists to stay close to their merged model while maintaining their own task performance. The paper also provides a theoretical analysis of the trade-off between specialization and mergeability. Experiments on vision and language benchmarks show improvements over several fine-tuning and merging baselines.

Critical strengths

1.The paper is well motivated and addresses an important problem in model merging. Improving compatibility during fine-tuning is a reasonable alternative to resolving conflicts after training.

2.The theoretical analysis provides an interesting explanation of why weak coupling can improve merging performance while having a relatively small effect on individual specialists.

3.The experimental results are strong. SCouT improves merging performance across different architectures and merging methods, and the analysis of coupling strength helps explain its behavior.

Critical weaknesses

1.The novelty over traditional soft parameter-sharing methods is unclear. With uniform averaging and isotropic coupling, the proposed regularizer is mathematically equivalent to pairwise L2 regularization between specialists. However, the paper does not include a direct comparison with standard Soft MTL methods.

2.SCouT requires specialists to be trained jointly with frequent parameter synchronization. This limits its applicability to scenarios where independently fine-tuned models are collected and merged later, which is a common setting in model merging.

3.The theoretical improvement relies on the condition χ > 0, but the paper does not sufficiently explain when this condition holds in practice. Moreover, the analysis assumes stationary specialists, while practical fine-tuning may not satisfy this assumption.

Questions for authors

please see weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer yrvq

We thank the reviewer for the positive assessment of our motivation, theory and experiments. All three concerns are shared with other reviewers and are addressed in detail in the General Response (GR); we summarize the key points below and answer the remaining questions.

W1: Novelty over soft parameter sharing

We agree that, with αk=1/K\alpha_k=1/K and Ak=IA_k=I, the SCouT coupling coincides with pairwise ℓ2\ell_2 soft sharing; the revision states this explicitly (Sec. 5.1). As detailed in GR, CC1, our contribution is twofold: (1) we show, to our knowledge for the first time, that this coupling makes specialists highly mergeable, and we characterize the resulting specialization--mergeability trade-off (Theorem 1); and (2) we generalize SCouT to a family of merge-aware algorithms, whose more targeted variants further improve performance (App. C.1-C.2).

Regarding a direct comparison with standard soft-MTL methods: since uniform, isotropic SCouT is pairwise ℓ2\ell_2 soft MTL, this comparison is given by our main SCouT rows (Table 1), with the coupling strength swept over a wide range (Fig. 4). What is new is evaluating it both before and after merging, which reveals the trade-off above.

W2: Joint training and frequent synchronization

It is true that SCouT requires synchronized fine-tuning, and therefore does not apply to completely independent training. SCouT instead targets settings where moderate communication is allowed, as in typical distributed and federated settings. As shown in GR, CC2, however, the synchronization need not be frequent: with periodic synchronization (App. C.3), 20×20\times fewer exchanges cost only 1.1 points in merged accuracy, and even with 20 exchanges SCouT outperforms all merger-friendly baselines under TA (76.72% vs. 69.14%). This makes SCouT well suited to distributed and federated settings with moderate communication budgets, where both strong specialists and a capable merged model are needed.

W3: Positivity of χ\chi and the stationarity assumption

As detailed in GR, CC3, a new lemma (Lemma 1 in the main text) guarantees χ>0\chi>0 under curvature-aware coupling whenever merging the independent specialists yields a higher risk than hard MTL. This is typically the case, e.g., 61.66% vs.
85.31% on ViT-B/32 (Table 1). For the isotropic coupling used in our experiments, the fitted post-merge slope gives χ≈9.9>0\chi\approx9.9>0 (Fig. 3) which verifies the assumption empirically.

Regarding the stationarity concern, Theorem 1 is a local result around stationary specialists. In practice, fine-tuning from a shared pretrained model with modern optimizers, here AdamW with a cosine schedule decaying to zero, reaches an approximately stationary regime: over T=2000T=2000 steps, the training loss and accuracy of all methods plateau. More importantly, we test the theory directly on these practical checkpoints: Fig. 3 (5 seeds) shows that the predicted asymmetry remains clearly visible, which provides empirical evidence that Theorem~1 describes the behavior of practical SCouT, beyond the idealized stationary setting.


We hope these revisions address the reviewer's concerns. Beyond its connection to soft parameter sharing, SCouT shows that weak coupling during fine-tuning substantially improves mergeability at almost no cost to specialization, a finding we believe is of broad interest to the model merging, multitask learning and distributed training communities. We would be grateful if the reviewer would consider updating the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their careful reading and constructive feedback. A revised paper is uploaded with all changes highlighted in green. The reviewers raised three common concerns, which we address jointly below: (CC1) the novelty of SCouT, since with αk=1/K\alpha_k=1/K and Ak=IA_k=I it reduces to soft parameter sharing (piYq, yrvq, zRF8); (CC2) its practical applicability, since SCouT requires frequent parameter synchronization during training (piYq, yrvq); and (CC3) the conditions under which the positivity assumption χ>0\chi>0 holds (piYq, yrvq).

CC1: Novelty relative to soft parameter sharing

We thank the reviewers for raising this point, and we agree with it. With αk=1/K\alpha_k=1/K and Ak=IA_k=I, the coupling induced by SCouT coincides with regularization-based soft parameter sharing. This fact has been stated explicitly and discussed in the revision immediately after the problem setting (Sec. 5.1). However, we would like to clarify our contributions relative to the literature on soft-parameter sharing and model mergeing:

(1) We show that parameter coupling, as in soft parameter sharing, is a simple yet effective way to make task-specific models substantially more compatible for merging. Because these models are merged into a single multitask model after fine-tuning, this raises a new question: the trade-off between mergeability and specialization. To our knowledge, the mergeability of soft-shared models, even in the isotropic case, has not been studied before; we investigate it theoretically in Theorem 1 and in experiments, and provide a detailed characterization of the specialization--mergeability trade-off.

(2) More generally, SCouT allows any training-time merge operator Merge⁡(ϑ)\operatorname{Merge}(\vartheta) and any coupling matrix AkA_k, giving a family of coupling methods that can be tuned to the merging strategy and available information. To illustrate this point, in the revision we added a comparison of Fisher, RegMean and AdaMerging operators (App. C.1, Table 4) and curvature-aware coupling (App. C.2, Table 5). Performance is highest when the training-time operator matches the test-time merger and when curvature enters the coupling matrix; cheaper and mismatched choices still give large gains over uncoupled baselines. Moreover, Theorem 1 applies to all of them for any continuously differentiable merge operator.

CC2: Practical applicability and communication cost

As pointed out by the reviewers, SCouT incurs additional communication cost in order to compute the merged model during training. However, given its significantly improved performance over independent fine-tuning (Table 1), we argue that this cost can be justified, depending on the constraints of a given system, such as in distributed or federated settings.

To explore the dependence of performance on communication resources, we have implemented a variant of SCouT, where the merged model is computed only periodically every ι\iota iterations (App. C.3), reducing communication by a factor of ι\iota. Our experiments (Table 6) show that SCouT's performance degrades gracefully as communication becomes less frequent. This suggests that SCouT is naturally robust to stale information and communication constraints. More elaborate schemes which rely on asynchronous or compressed computation of the merged reference model are certainly possible and an interesting direction for future research.

CC3: Positivity of χ\chi

Indeed, χ>0\chi>0 is not automatic. The revision addresses this both theoretically and empirically.

We add Lemma 1 in the main text, which shows when the local task risks are quadratic around the limiting point θo\theta^o, curvature-aware coupling (Ak=HkA_k=H_k) guarantees positivity by:

\Big[\sum_{k}J_k(\theta^o_{\mathrm{merge}})-\sum_{k}J_k(\theta^{\mathrm{MTL}})\Big]\ \ge 0.$$ When using isotropic coupling ($A_k=I$), as in the majority of our main experiments, the lemma does not apply and $\chi>0$ is not guaranteed in general. In that sense, $\chi$ can be viewed as a measure of the (mis)match between the merge-aware coupling term and the actual merging operation. A positive $\chi$ indicates that the objectives are sufficiently well-matched to guarantee a boost in performance. Empirically, our experiments show that for $\alpha_k=1/K$ and $A_k=I$ and the classical model architectures and data sets used, $\chi$ is indeed positive: The first-order fit of the post-merge risk in Fig.~3 yields a slope of $-9.9$, i.e., $\chi\approx9.9>0$.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.