- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to improve the mergeability of task specialists while preserving their individual performance. SCouT adds a parameter penalty between each specialist and a prescribed merged model during fine-tuning. A local analysis gives a conditional regime where merged risk improves to first order in the coupling strength while specialist risk changes to second order. Experiments with CLIP and RoBERTa show a useful tradeoff across coupling strengths and several post-hoc merging rules.
Critical strengths
-
The empirical tradeoff is useful and well supported. Table 1 raises the ViT-B/32 merged accuracy from 61.66 to 81.27 while keeping specialist accuracy at 87.83, compared with 88.62 for independent fine-tuning. Table 2 shows gains across five merging rules, and Appendix D extends the result to GLUE. The paper also measures communication and memory costs in Appendix E. These results give practical evidence that mild coordination can retain useful specialists and improve their combined model.
-
The paper connects its method to an interpretable local risk analysis. Section 4 states the positive alignment condition needed for the weak-coupling advantage, and Figures 3 and 4 examine both risk response and the broader performance tradeoff. This conditional explanation helps understand why mergeability can improve before specialist performance drops substantially.
Critical weaknesses
-
I find the distinction from standard soft parameter sharing unclear in the implemented setting. Equation 16 uses the parameter mean as the merged model, so its coupling is exactly a pairwise squared-distance penalty up to a constant factor. Section 2.2 acknowledges this close connection. I do not see a new regularizer arising from the change in downstream goal alone, and I currently understand the main contribution as the merging perspective and its validation. I would find the originality easier to assess if the authors stated this equivalence explicitly.
-
My concern with Theorem 1 and Appendix A.3 is how the stationary result relates to the actual optimizer. Equations 55 to 62 establish a zero projected gradient on the range of the independent specialists' Hessians. A nonzero component of the full gradient can still remain in the Hessian null space. Algorithm 1 does not impose the range constraint. I would like the authors to state the restricted stationary result explicitly and explain its connection to the unconstrained training trajectory. I regard this as a fixable issue in the scope of the claim, but the risk asymmetry is a central contribution and needs this connection.
Questions for authors
-
I would like the authors to state the equivalence between the uniform isotropic setting in Equation 16 and pairwise soft parameter sharing, and explain which part of the contribution goes beyond that existing objective.
-
I would like to know whether Theorem 1 concerns stationary points of the reduced objective on the Hessian range. I would also like the authors to identify the condition connecting this branch to Algorithm 1 when the Hessians have null spaces. A corrected statement would help resolve my concern.
2 comments
I appreciate the authors' detailed response and find the points well-reasoned. The clarifications have largely addressed my primary concerns, so I am pleased to maintain my score and lean towards recommending acceptance.
Response to Reviewer zRF8
We thank the reviewer for the positive and constructive assessment, and for the precise technical comments, which helped us sharpen both the positioning and the theory of the paper. The first concern is shared with other reviewers and is addressed in detail in the General Response (GR); we summarize it below and answer the second concern, which is specific to this review, in full.
W1 / Q1: Equivalence with soft parameter sharing
The reviewer is correct in pointing out the equivalence with soft parameter sharing, and the revision now states this fact explicitly (Sec. 5.1, App. C.1). As detailed in GR, CC1, our contributions beyond this objective are (1) to show, to our knowledge for the first time, that the soft coupling/regularization makes specialists highly mergeable, and to characterize the resulting specialization-mergeability trade-off (Theorem 1); and (2) as the reviewer anticipated, SCouT can generalize to a family of merge-aware algorithms with general training-time merge operators and coupling matrices. These variants are evaluated in the revision (App. C.1-C.2).
W2 / Q2: Stationarity in Theorem 1 and connection to Algorithm 1
We thank the reviewer for this careful observation, which led us to revise Theorem 1 and to connect the theory more closely to the algorithm.
The revised Theorem 1 no longer needs the range restriction. Assuming positive definite Hessians Hk≻0, Theorem 1 gives a unique stationary point of the unconstrained objective Fλ,
ϑλ=ϑo−λH−1∇ϑC(ϑo)+O(λ2), which is exactly the objective optimized by Algorithm 1. For singular Hessians Hk⪰0, a new remark (App. A.4) states the range-restricted result explicitly.
We acknowledge that neither result fully settles the case of singular Hessian H: Theorem 1 requires a positive definite Hessian Hk≻0, while the remark holds only on the range of H. However, assuming an invertible (or damped) Hessian is standard in local perturbation analyses of deep networks, e.g., influence functions use H+γI (Koh & Liang, ICML 2017; Grosse et al., 2023). More importantly, the contribution of Theorem 1 is not an exact formula for ϑλ, but the mechanism and scaling orders it reveals: weak coupling improves the merged risk at first order while costing specialist risk only at second order. These orders are what the paper relies on, and they are confirmed empirically (Fig 3, Fig 4 and Table 1).
Beyond the revised theorem, the revision strengthens the link between Theorem 1 and practical SCouT on two further fronts. First, theoretically, a new lemma (Lemma 1 in the main text) shows that, in the local quadratic model, the alignment coefficient χ of Theorem 1, which governs the first-order post-merge improvement, is guaranteed to be positive under curvature-aware coupling. Second, empirically, Fig. 3 tests the predictions on checkpoints of Algorithm 1 (5 seeds): the post-merge risk decreases linearly, with a fitted slope giving χ≈9.9>0, while the specialist risk grows only quadratically. The asymmetry predicted by Theorem 1 thus carries over to the actual training trajectory.
We hope these revisions address the reviewer's concerns. As the reviewer noted, soft coordination during fine-tuning can retain useful specialists while improving their merged model. We believe this opens a new direction: coordinated fine-tuning in settings where moderate communication is allowed, bridging model merging, multitask learning and distributed training. We hope it will be of broad interest to these communities, and we would be grateful if the reviewer would consider updating their score.